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PERFORMANCE  ASSESSMENT  CAPABILITY  FEASIBILITY  STUDY 

Introduction 

This  report  presents  the  results  and  recommendations  of  the  performance  assessment 
capability  feasibility  study  conducted  by  the  Army  Research  Institute  (ARI)  Ft.  Bliss  Field 
Unit  for  the  Directorate  of  Training  Development  (DOTD)  at  the  US  Army  Air  Defense 
Artillery  School  (USAADASCH).  Previously,  ARI  had  developed  a  performance  assess¬ 
ment  capability  (PAC)  for  a  research  environment  and  had  implemented  it  on  a  Patriot  tac¬ 
tical  operations  simulator.  The  objective  of  the  study  was  to  determine  the  feasibility  of 
using  the  ARI-developed  PAC  as  the  basis  for  improved  operator  performance  evaluation 
in  the  Patriot  training  environment. 

Hie  Problem 

The  need  for  improved  Patriot  operator  performance  assessment  was  officially  recog¬ 
nized  at  the  March  1989  General  Officer  Echelon  Above  Corps  Review.  The  poor  evalua¬ 
tion  capabilities  of  Patriot  training  simulators  were  cited  as  a  major  obstacle  to  conducting 
effective  Patriot  operator  training.  Action  Item  II-9  of  the  Review  states:  "Automatic  scor¬ 
ing  has  recognized  deficiencies.  The  Patriot  community  must  work  this  as  a  joint  effort." 

'l  he  Patriot  high  altitude  air  defense  artillery  weapon  system  is  described  as  the 
Army’s  first  fully  automated  weapon  system.  It  typically  operates  with  two  operators  at 
each  battery  fire  unit  and  two  at  battalion.  The  operator  functions  are  broken  into  the 
friendly  protector,  who  is  responsible  for  verifying  and  overriding  system-assigned  aircraft 
identifications,  and  the  weapons  controller,  who  initiates  aircraft  engagements.  The  sys¬ 
tem  computer  and  the  training  computer,  in  addition  to  their  primary  functions,  can  be  used 
to  collect  data  and  calculate  and  report  scores.  However,  the  sheer  power  of  automation 
alone  does  not  mean  that  the  scoring  is  adequate,  a  criticism  that  applies  not  only  to  Patriot 
but  to  the  majority  of  automated  systems  today. 

The  principal  Patriot  scoring  deficiency  is  the  composite  score  used  to  assess  tactical 
performance  on  both  the  Operations  Tactical  Trainer  (OTT)  and  the  Troop  Proficiency 
Trainer  (TPT).  The  OTT  is  used  by  USAADASCH  to  provide  procedures  training  at  the 
institution  or  schoolhouse  and  is  used  by  the  32nd  Army  Air  Defense  Command  (AAD- 
COM)  to  provide  individual  and  collective  tactics  training.  The  TPT  is  embedded  in  the 
tactical  system  and  is  used  for  maintaining  individual  and  collective  tactical  proficiency. 

The  composite  score  used  on  both  devices  is  a  weighted  combination  of  asset  damage,  hos¬ 
tile  attrition,  and  missile  utilization. 

Several  problems  are  associated  with  the  composite  score.  For  one,  the  composite 
score  does  not  provide  diagnostic  information  that  can  help  a  student  or  instructor  pinpoint 
and  remedy  a  performance  problem.  If  an  operator  obtains  a  low  score,  it  can  be  at¬ 
tributable  to  poor  performance  in  one  or  all  of  the  areas  incorporated  into  the  score.  The 
composite  score  is  limited  further  in  that  it  considers  only  hostile  aircraft.  If  an  operator 
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destroys  all  friendly  aircraft  while  destroying  all  hostile  aircraft  and  preventing  asset 
damage,  he  or  she  can  still  receive  a  satisfactory  composite  score.  Clearly,  fratricide 
should  be  included  in  any  assessment  of  operator  and  system  performance.  Finally,  errors 
in  software  algorithms  have  led  to  unreliable  calculation  of  the  composite  score,  which  has 
eroded  confidence  in  it. 

In  addition  to  the  composite  score  used  to  evaluate  tactical  performance,  the  OTT  has 
a  procedural  evaluation  capability.  In  this  evaluation,  critical  student  sw'itch  actions  are 
recorded  during  an  air  battle  scenario  and  later  compared  to  those  of  an  expert.  Points  are 
deducted  from  the  student’s  score  when  his  or  her  switch  actions  deviate  by  more  than  a 
specified  time  margin  from  the  expert’s.  A  shortcoming  of  this  approach  is  that  a  student 
can  choose  to  process  aircraft  in  a  different  order  than  the  expert  and  still  achieve  an  equal¬ 
ly  effective  outcome  in  terms  of  asset  defense,  attrition,  etc.  However,  because  the  order 
in  w'hich  the  student  processes  aircraft  is  different,  the  sequence  of  switch  actions  is  also  dif¬ 
ferent  from  the  expert's.  Consequently,  the  student  receives  a  low  score  on  procedural 
performance  in  spite  of  the  fact  that  overall  tactical  performance  is  as  good  as  the  expert’s. 

The  Solution 

DOTD  was  tasked  with  developing  a  solution  to  the  Patriot  performance  assessment 
problem.  DOTD  devised  a  three-phased  solution:  Phase  1:  use  senior  instructors  and  sub¬ 
ject  matter  experts  to  provide  "over-the-shoulder"  performance  assessment,  a  workable,  but 
labor-intensive,  short-term  solution;  Phase  2:  develop  and  implement  a  baseline, 
automated  PAC  that  provides  immediate,  high  level  feedback  for  instructors,  evaluators, 
and  students  and  off-line,  detailed  performance  assessment  for  course  developers,  analysts, 
and  training  management;  and  Phase  3:  enhance  and  extend  the  baseline  PAC  to  provide  a 
more  fully  automated,  "smart"  PAC  that  provides  immediate,  detailed  diagnostics  and  criti¬ 
ques. 


DOTD  immediately  implemented  Phase  1  of  the  performance  assessment  solution. 

As  a  first  step  in  Phase  2,  DOTD  and  ARI  entered  into  an  agreement  to  investigate  the 
feasibility  of  transferring  all  or  part  of  the  PAC  previously  developed  by  ARI  to  the  OTT 
and  TPT.  The  ARI  PAC  is  described  in  the  following  section.  The  method  for  conduct¬ 
ing  the  study,  the  results,  and  the  recommendations  are  presented  in  subsequent  sections. 

Overview  of  the  ARI  PAC 

Under  the  High  Altitude  Air  Defense-Console  Operator  Performance  work  unit,  ARI 
developed  a  performance  assessment  capability  (Allender,  1987;  Allender  &  Brett,  1988) 
based  on  earlier  conceptual  work  (Hawley,  Brett,  &  Chapman,  1982;  Hawley,  Howard,  & 
Martellaro,  1982).  The  ARI  PAC  is  characterized  by  three  features.  ( 1 )  It  is  descriptive 
of  both  system  and  operator  performance.  It  consists  of  four  levels  of  performance 
measures.  The  top  level  describes  the  combined  contribution  of  the  operators,  hardware, 
and  software.  Each  successive  level  provides  a  more  fine-grained  description  of  the  con¬ 
tribution  of  individual  operators.  (2)  The  PAC  captures  the  outcomes  of  decision  making. 
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It  is  not  derived  from  a  traditional  task  analytic  approach  that  stresses  rigid  sequences  of  ac¬ 
tions.  Rather,  it  examines  decision  making  with  respect  to  critical  time  window's  and  asses¬ 
ses  how  decisions  contribute  to  mission  outcomes.  (3)  The  PAC  supports  performance 
diagnosis.  Through  the  multiple  levels  of  performance  measures,  specific  performance 
problems  and  successes  can  be  pinpointed  and  linked  directly  to  mission  outcomes. 

The  PAC  was  implemented  on  the  Patriot  Tactical  Operations  Simulator  (PTOS) 
operated  by  the  Directorate  of  Combat  Developments  (DCD)  at  USAADASCH.  The 
PTOS  is  a  realtime,  high-fidelity  mission  simulator  of  the  battery  fire  unit  and  the  battalion 
operator  consoles  of  the  Patriot  missile  system.  The  PAC  software  consists  of  a  realtime 
data  collection  module  and  post-scenario  data  reduction,  summarization,  and  performance 
measures  generation  modules.  It  is  a  working  system  and  its  effectiveness  in  describing 
and  assessing  operator  performance  has  been  demonstrated  in  several  studies. 

Figure  1  presents  the  structure  of  the  PAC.  As  indicated  above,  the  PAC  assesses 
performance  at  four  levels.  The  top  level  consists  of  mission  performance  measures 
(MPMs)  that  assess  overall  system  performance  on  the  standard  air  defense  missions  of 
point  defense  (defense  of  assets)  and  area  defense  (attrition).  Also  included  at  this  level 
are  measures  that  assess  the  critical  system  objectives  of  fratricide  control  (friendly  protec¬ 
tion)  and  effective  missile  utilization  (resource  conservation).  MPMs  reflect  total  system 
performance  and  include  the  contributions  of  the  operators,  the  hardware,  and  the  software. 

The  next  level  of  assessment  is  called  function  performance  measures  (FPMs). 

FPMs  assess  the  operators’  ability  to  perform  target  engagement  (weapons  controller)  and 
target  identification  (friendly  protector)  functions.  For  each  operator  function,  measures 
that  assess  critical  dimensions  of  function  performance  are  provided.  For  the  friendly 
protector  function,  for  example,  the  dimensions  are  thoroughness  (were  all  of  the  aircraft 
identified?),  accuracy  (were  the  correct  identifications  assigned  to  each  aircraft?),  and 
timeliness  (were  the  friends  identified  before  they  could  be  engaged  and  the  hostiles  before 
they  could  attack  assets?).  The  measures  that  reflect  these  dimensions  are  percent  aircraft 
identified,  percent  identified  correctly,  and  percent  identified  late,  respectively.  Essential¬ 
ly,  FPMs  focus  on  the  outcomes  of  engagement  and  identification  decision  making  and  pro¬ 
vide  an  intermediate  level  of  diagnostics. 

With  FPMs,  the  unique  operator  contribution  to  mission  performance  is  isolated. 
Consequently,  computation  of  FPMs  includes  only  those  aircraft  in  a  scenario  that  the 
operator  should  have  acted  upon  (identified  or  engaged).  In  a  given  scenario,  for  example, 
all  hostiles  might  not  come  into  range.  Therefore,  the  maximum  possible  attrition  score  is 
less  than  100%.  The  FPM  percent  hostiles  killed,  however,  considers  only  those  hostiles 
that  do  come  in  range.  Thus,  a  percent  hostiles  killed  value  of  100  indicates  the  operator 
contributed  maximally  to  the  attrition  mission. 

Task  performance  measures  (TPMs),  the  third  level  of  measures,  assess  actions  that 
t  <•:  ...  tribute  to  successful  function  performance.  There  are  two  types  of  TPMs: 
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Figure  1.  ARI  PAC  structure. 

switch-based  and  alert-based.  Switch-based  TPMs  primarily  assess  the  operator’s  use  of 
hooks  and  the  Identification  Friend  or  Foe  (IFF)  system.  Hooking,  for  example,  is  impor¬ 
tant  because  it  is  a  prerequisite  action  for  engaging  or  identifying  aircraft.  Scores  of  less 
than  100%  of  aircraft  hooked  can  help  to  explain  poor  identification  and  engagement 
functioning.  Alert-based  TPMs  assess  an  operator’s  efficiency  at  processing  alert  mes¬ 
sages.  Failure  to  efficiently  process  alerts  can  result  in  critical  information  being  lost  or  ex¬ 
cessively  delayed. 

The  lowest  level  of  performance  assessment  diagnostics  is  the  detailed  performance 
histories  (DPHs).  A  sample  is  provided  in  Figure  2.  DPHs  provide  a  second-by-second 
description  of  what  the  operator  and  the  system  were  doing  on  each  aircraft  in  a  scenario. 

A  time-line  format  is  used  and  six  classes  of  information  are  provided:  (1)  the  identifica¬ 
tion  and  (if  applicable)  engagement  windows;  (2)  hook  counts  and  durations;  (3)  operator 
actions  other  than  hooks;  (4)  .he  aircraft’s  identification  history;  (5)  Patriot  system  events 
such  as  time  to  launch  release  reaching  zero,  the  aircraft  entering  the  to-be-engaged-queue, 
and  missile  launches;  and  (6)  aircraft  events  (track  events)  such  as  track  number  changes 
and  IFF  responses.  With  these  detailed  data,  where  operator  attention  was  focused  at  a 
particular  point  in  the  scenario  and  what  actions  were  being  performed  can  be  determined. 
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Figure  2.  Sample  DPH. 

This  micro-level  information  can  be  related  directly  back  to  specific  decisions  (e.g.,  iden¬ 
tification  assignments)  and  patterns  of  performance  problems  can  be  detected. 

As  indicated  at  the  beginning  of  the  section,  the  concept  of  decision  windows  is  essen¬ 
tial  to  the  assessment  strategy  employed  by  the  PAC.  For  each  aircraft  scripted  in  a 
scenario,  an  identification  window  must  be  established  and,  if  the  flight  is  scripted  to  be  hos¬ 
tile,  an  engagement  window  must  be  established.  Essentially,  establishing  identification 
and  engagement  windows  is  a  matter  of  formulating  the  rules  that  define  window  start  and 
end  points  and  applying  them  to  each  aircraft  flight  path  in  a  scenario  to  determine  when 
those  points  occur  for  each  aircraft.  Three  factors  influence  window  definition  in  relation 
to  an  aircraft  flight  path:  (1)  the  design  of  the  defense  (e.g.,  placement  of  volumes  and  cor¬ 
ridors),  (2)  the  tactical  standing  operating  procedure  (TSOP)  and  its  associated  rules  and 
procedures  that  control  conduct  of  the  air  battle  (e.g.,  weapons  control  status  or  identifica¬ 
tion  weight  set),  and  (3)  Patriot  system  capabilities  (e.g.,  how  far  can  it  shoot?,  how  fast  is 
the  missile?,  how  well  can  the  radar  "see"  in  jamming?). 
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As  an  example,  a  typical  definition  of  an  engagement  window  defines  window  start  as 
the  first  point  in  time  at  which  both  a  high  probability  of  kill  (Pk)  exists  and  the  aircraft  is 
considered  engageable  under  the  TSOP  (i.e.,  aircraft  is  hostile  under  weapons  tight  or  hos¬ 
tile  or  unknown  under  weapons  free).  System  capabilities  determine  the  point  at  which  a 
high  Pk  engagement  is  possible.  Flight  path  scripting  in  relation  to  the  defense  layout  and 
TSOP  determines  what  an  aircraft’s  identification  should  be,  and,  hence,  when  it  will  attain 
an  engageable  identification.  The  end  of  the  window  is  defined  as  the  point  in  the 
scenario  at  which  an  engagement  must  be  initiated  so  that  intercept  occurs  before  the 
aircraft  can  penetrate  an  asset.  This  point  is  driven  by  the  interaction  of  the  aircraft  path 
with  the  missile  capabilities  (i.e.,  speed)  and  Fire  unit  placement  relative  to  assets  in  the 
defense  layout. 

Identification  and  engagemen:  windows  have  to  be  specified  prior  to  using  the  PAC  to 
evaluate  performance  on  a  scenario.  The  PAC  compares  operator  actions  and  times  with 
the  window  specifications  for  each  aircraft  in  a  scenario  in  order  to  calculate  the  window- 
based  FPM’s  (e.g.,  percent  late  identifications,  percent  late  engagements).  In  summary, 

(  AC  decision  windows  provide  a  means  of  evaluating  operator  performance  that  reflects 
the  unique  decision  demands  of  any  scenario. 

In  addition  to  providing  a  basis  for  assessing  the  timeliness  of  decision  making,  PAC 
windows  also  provide  a  means  for  controlling  scenario  difficulty.  By  varying  the  number  of 
windows  (number  of  aircraft),  the  average  length  of  windows,  and  the  amount  of  overlap  be¬ 
tween  windows,  the  difficulty  of  a  scenario  is  varied.  Scenarios  with  a  small  number  of 
aircraft,  long  windows,  and  little  overlap  between  windows  form  the  low  end  of  the  difficul¬ 
ty  continuum.  The  high  end  of  the  continuum  is  composed  of  scenarios  with  large  num¬ 
bers  of  aircraft  and  short  windows  that  overlap  considerably. 

Method 

The  PAC  feasibility  study  consisted  of  four  major  tasks:  (1)  Specify  ARI  PAC  perfor¬ 
mance  measures  and  other  features  required  and  desired  by  OTT  and  TPT  users,  (2) 

Specify  data  collection  requirements  to  support  PAC  measure  computation,  (3)  Specify  and 
evaluate  alternative  PAC  implementation  schemes,  and  (4)  Prepare  report  of  study  results. 
The  method  and  procedures  for  tasks  1  through  3  are  described  below. 

Task  1:  Specify  ARI  PAC  Performance  Measures 

There  were  two  sub-tasks  in  this  task.  The  first  was  to  select  which  of  the  ARI  PAC 
performance  measures  were  required  or  desired  by  OTT  and  TPT  users.  The  ARI  PAC 
contains  a  large  number  of  measures  and  it  was  expected  that  only  a  subset  would  be  of  in¬ 
terest  to  the  training  community.  Further,  it  was  recognized  that  there  are  a  number  of  dif¬ 
ferent  OTT  and  TPT  user  groups  (i.e.,  students,  instructors,  evaluators,  courseware 
developers,  training  analysts,  etc.)  and  that  PAC  information  or  feedback  requirements 
would  vary  from  group  to  group.  To  ensure  that  the  selection  of  ARI  PAC  measures 
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would  be  sensitive  to  the  unique  information  and  feedback  requirements  of  different 
groups,  separate  rating  sessions  were  held  for  each  OTT  and  TPT  user  group. 

Each  rating  session  began  with  introductory  comments  about  the  purpose  and  objec¬ 
tives  of  the  feasibility  study.  Next,  an  in-depth  description  of  the  ARI  PAC  was  given 
w-hich  included  definitions  of  each  of  the  measures  (see  Appendix  A).  Following  the 
briefing,  the  users  provided  ratings  on  the  ARI  PAC  measures.  Within  each  of  the  four 
PAC  levels  (MPM,  FPM,  TPM,  and  DPH),  users  rated  each  individual  measure  in  terms  of 
its  usefulness  to  them  in  their  jobs.  Next,  they  rated  each  of  the  four  levels  on  how  quickly 
they  wanted  the  measures  reported  (timeliness)  and  on  how  often  they  wanted  the 
measures  (frequency).  They  also  indicated  in  what  formats  they  wanted  the  PAC  data 
(e.g.,  printouts,  interactive  data  display,  scenario  replay  with  a  "smart"  critique).  Finally, 
extra  pages  were  provided  for  any  comments  they  wanted  to  make  about  the  PAC  and  the 
study.  See  Appendix  B  for  a  copy  of  the  rating  form. 

The  second  sub-task  was  to  identify  tasks,  beyond  those  assessed  by  the  ARI  PAC, 
that  should  be  evaluated  by  a  PAC  for  an  OTT  and  a  TPT.  It  was  recognized  that  whereas 
the  ARI  PAC  assesses  only  engagement  and  identification  task  performance,  a  broader 
range  of  tasks  would  be  of  interest  to  OTT  and  TPT  users.  The  process  of  identifying  addi¬ 
tional  tasks  was  broken  down  into  two  s.  ’ps.  First,  14E  (Patriot  officer)  and  24T  (Patriot 
enlisted)  soldier  task  lists  w'ere  reviewed  and  a  list  of  candidate  tasks  selected.  Criteria  for 
inclusion  in  the  candidate  task  list  was  broad.  Basically,  any  task  involved  in  preparing  the 
system  for  action  and  conducting  an  air  battle  was  included.  Tasks  in  other  areas  such  as 
maintenance  and  march  order  and  emplace  were  excluded.  See  Appendix  C  for  the  can¬ 
didate  task  iist.  Next,  a  working  group  was  convened  with  representatives  from  each  of  the 
OTT  and  TPT  user  groups.  They  reviewed  the  candidate  task  list  and  provided  guidance 
on  additional  tasks  to  be  included  in  an  OTT  and  TPT  PAC. 

Task  2;  Specify  Data  Collection  Requirements 

In  Task  1,  performance  measures  for  an  OTT  and  TPT  PAC  were  specified.  In  Task 
2,  the  da^a  elements  required  to  compute  those  performance  measures  were  defined.  A 
performance-measure-by-required-data-element  matrix  was  developed  in  three  steps. 

First,  formulae  for  computing  the  PAC  measures  were  stated.  Second,  the  data  elements 
required  to  compute  each  measure  were  specified.  Finally,  the  matrix  cells  were  filled  to 
indicate  which  data  elements  were  required  by  which  measures.  A  second  matrix  was  also 
developed  to  give  DOTD  an  indication  of  the  extent  to  which  the  current  OTT  could  sup¬ 
port  PAC  implementation.  In  this  matrix  ARI  PAC  data  elements  were  cross  referenced 
with  OTT  data  messages  and  variables.  In  order  to  develop  this  matrix,  an  analysis  of  OTT 
software  was  conducted  to  identify  OTT  variables  and  data  that  were  the  equivalents  of 
PAC  data  elements. 
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The  objective  of  this  task  was  to  explore  various  means  of  implementing  the  ARI 
PAC  in  OTT  and  TPT  environments.  Long  term,  it  meant  providing  functional  design 
specifications  that  would  permit  incorporation  of  a  PAC  into  next  generation  OTTs  and 
TPTs.  Near  term,  it  meant  exploring  ways  of  implementing  a  PAC  on  the  existing  OTT. 
Two  alternatives  were  developed  by  combining  hardware  and  software  configurations  that 
could  be  used  to  collect  data  and  calculate  performance  measures.  The  evaluation  of  alter¬ 
natives  was  driven  by  five  factors:  (1)  relative  estimated  software  development  costs 
(specified  at  a  gross  level),  (2)  relative  estimated  hardware  costs,  (3)  potential  impact  on 
simulation  reliability,  (4)  potential  impact  on  simulation  realtime  performance,  and  (5) 
ability  to  interface  with  future  system  upgrades  and  modifications.  In  addition  to  exploring 
impiementation  alternatives,  suggestions  for  the  OTT  and  TPT  PAC  user  interface  were 
made.  These  included  the  layout  of  screens  used  to  obtain  PAC  information  and  the  na¬ 
ture  of  the  user  interface. 

Resul 

Results  of  Task.. J:  Specify  _P  AC  Performance  Measures 

Results  of  sub-task  1.1:  Select  ARI  PAC  performance  measures.  Seventy-six  persons 
from  13  user  groups  (12  OTT  and  TPT  user  groups  and  one  Hawk  user  group)  participated 
in  the  ARI  PAC  rating  sessions.  Table  1  lists  the  groups  that  participated.  As  evidenced 
in  the  table,  virtually  all  US  Army  organizations  actively  involved  in  developing,  providing, 
and  analyzing  Patriot  training  both  in  the  US  and  Europe  participated  in  rating  sessions. 
Also,  four  field  units  from  the  US  and  Europe  participated.  In  short,  OTT  and  TPT  user 
groups  were  well  represented  in  the  rating  sessions.  The  Hawk  Department  was  included 
because  it  was  recognized  that  many  of  the  ARI  PAC  measures  were  directly  applicable  to 
the  Hawk  environment  and  that  Hawk  would  be  one  of  the  next  arenas  for  PAC  implemen¬ 
tation.  Formal  briefings  on  the  PAC  were  also  provided  to  the  Army  Training  and 
Doctrine  Command  System  Manager  for  Patriot  and  to  the  West  German  Air  Force,  but  no 
formal  ratings  were  obtained. 

Table  2  presents  average  usefulness  ratings  for  measures  in  the  fear  different  levels  of 
the  PAC  broken  out  by  user  group.  The  user  groups  are  further  organized  into  four  larger 
groups  according  to  similarity  of  function:  training  analysts,  trainers  (those  who  develop  and 
deliver  training),  field  personnel,  and  the  Hawk  Department.  Mean  usefulness  ratings  are 
also  presented  for  each  of  the  larger  groups.  In  the  table,  the  higher  the  number,  the  more 
useful  the  measures  were  rated  (1  =  not  at  all  useful,  5  =  very  useful). 

Reviewing  the  ratings,  several  results  are  apparent.  (1)  Across  user  groups,  the 
ratings  were  high:  almost  all  ratings  exceeded  3.0.  A  rating  of  3.0  indicates  that  a  measure 
is  considered  somewhat  useful.  In  short,  most  of  the  user  groups  want  most  of  the  ARI 
PAC  measures.  (2)  Most  user  groups  rated  the  MPMs  as  the  most  useful  of  all  the  levels, 
showing  that  a  somewhat  higher  premium  is  placed  on  the  measures  that  assess  overall  mis- 
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Table  1 


List  of  OTT  and  TPT  User  Group  Organizations  that  Participated  in  the  ARI  PAC  Rating 
Sessions 


Participants 


Group _ Number  of 

DOTD  2 

Directorate  of  Evaluation  and  Standardization/ 

Concepts  and  Studies  Division  (DESCSD)  4 

Patriot  Training  Dept.  (14E)  8 

Patriot  Training  Dept.  (24T)  7 

Patriot  Training  Dept.  (Devices)  5 

Combined  Arms  and  Tactics  Dept.  (CATD)  3 
32  AADCOM  OTT  4 

32  AADCOM  Training  &  Eval  Team  4 

1 1TH  ADA  BDE  13 

6TH  ADA  BDE  3 

4/43  ADA  (32  AADCOM)  2 

4/7  ADA  (32  AADCOM)  18 

Hawk  Department _ 1 _ 3 

TOTAL  76 


sion  performance  compared  to  the  more  detailed  levels.  The  analysts  rated  the  DPHs  the 
lowest  of  the  four  levels,  whereas  the  trainers  and  the  field  unit  personnel  rated  the  TPMs 
the  lowest.  (3)  Ratings  provided  by  the  analysts  were  consistently  high  across  all  levels: 
the  lowest  mean  observed  is  4.33  for  the  DPHs.  Also,  their  ratings  were  generally  higher 
than  those  provided  by  trainers  and  field  unit  personnel,  reflecting  their  heavy  reliance  on 
such  data  to  pinpoint  performance  problems  and  relate  them  to  system  performance.  (4) 
Comparison  of  the  ratings  of  trainers  with  personnel  in  field  units  shows  little  difference  in 
the  usefulness  attached  to  the  different  levels  of  measures.  It  had  been  expected  that 
trainers  would  want  many  of  the  PAC  measures  because  they  must  be  able  to  diagnose  per¬ 
formance  problems  in  order  to  provide  remediation.  What  is  of  interest  here  are  the 
ratings  of  personnel  from  the  field.  Not  only  do  they  want  the  high  level  mission  and  func¬ 
tion  feedback,  they  want  the  detailed  feedback  as  well.  This  seems  to  reflect  a  recognition 
by  field  personnel  that  detailed  feedback  is  required  to  maximize  and  fine  tune  perfor¬ 
mance. 

A  review  of  the  mean  usefulness  ratings  for  each  individual  ARI  PAC  measure  by 
user  group  data  (see  Appendix  D)  supports  and  extends  the  presentation  of  results  in  Table 
2.  Very  few  individual  measures  had  mean  user  group  ratings  less  than  3.0.  However, 
there  were  two  types  of  measures  that  were  consistently  rated  lower  than  others.  These 
were  the  function  performance  measures  of  (1)  the  percentage  of  identifications  and 
enablements  that  are  early,  that  is,  before  the  start  of  the  associated  windows  and  (2) 
average  time  delays  to  identify  and  to  engage  targets  (i.e.,  total  time  after  window  start,  as 
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Table  2 

Mean  Usefulness  Ratings  of  AR1  PAC  Measures  in  a  Level  by  User  Group 


Group 

MPMs 

FPMs 

TPMs 

— Il  1  II 

Analysts 

DOTD 

5.00 

4.36 

4.50 

4.00 

DESCSD 

4.78 

4.86 

4.63 

4.20 

Means 

4.89 

4.68 

4.57 

4.33 

Trainers 

Pat  Trn  Dept.  (14E) 

4.28 

4.00 

3.88 

3.98 

Pat  Trn  Dept.  (24T) 

2.68 

3.06 

2.65 

3.37 

Pat  Trn  Dept.  (Devices) 

4.50 

4.18 

3.72 

2.97 

CATD 

4.68 

4.06 

3.95 

4.35 

32  AADCOM  OTT 

4.70 

4.35 

4.23 

4.40 

32  AADCOM  Trn  &  Eval 

4.53 

4.33 

4.22 

4.32 

Means 

4.22 

3.91 

3.67 

3.81 

Field  Unit  Personnel 

11THBDE 

4.53 

4.19 

3.95 

4.03 

6TH  BDE 

4.25 

3.93 

3.40 

3.80 

4/43  ADA  (32  AADCOM) 

4.75 

3.76 

4.35 

4.42 

4/7  ADA  (32  AADCOM) 

4.53 

3.91 

3.45 

3.58 

Means 

4.52 

4.01 

3.67 

3.80 

Hawk  Department 

4.65 

3.76 

2.44 

3.38 

Grand  Means* 

437 

4.01 

3.70 

3.83 

’Throughout  grand  or  overall  means  are  based  on  individual  ratings,  not  group  means. 


differentiated  from  percent  late,  or  after  window  end).  Average  ratings  for  these  FPMs 
were  3.6  versus  4.2  for  all  the  other  FPMs.  Also  of  note  is  that  the  Hawk  user  group  gave 
the  lowest  possible  ratings  to  the  alert-based  TPMs.  This  is  because  alerts  in  the  Hawk  sys¬ 
tem  are  not  at  all  like  alerts  in  the  Patriot  system  and,  therefore,  the  alert-based  TPMs  were 
meaningless  in  the  Hawk  context. 

Table  3  presents  mean  ratings  of  expected  frequency  of  use  (1  =  quarterly  or  less,  3 
=  weekly,  5  =  several  times  a  day)  and  required  timeliness  (1  =  a  week  or  longer,  3  =  a 
day,  5  =  10  minutes  or  less)  for  the  different  levels  of  PAC  measures.  The  most  notable 
characteristics  of  these  ratings  are  the  consistency  across  levels  within  user  groups  and  the 
variability  between  user  groups.  Within  groups,  the  frequency  ratings  were  similar  across 
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Table  3 

Mean  Ratings  of  Required  Frequency  and  Timeliness  of  Data  from  ARI  PAC  Levels  by 
User  Group 


MPMs  FPMs  TPMs  DPHs  Overall 

Frequency  Frequency  Frequency  Frequency  Frequency 

Group _ Timeliness* _ Timeliness _ Timeliness  Timeliness _ Timeliness 

Analysts 


DOTD 

1.5 

1.8 

1.5 

2.0 

1.7 

1.5 

1.5 

1.5 

1.5 

1.5 

DESCSD 

1.3 

1.3 

1.3 

1.3 

1.3 

2.8 

2.8 

2.8 

2.8 

2.5 

Trainers 

Pat  Tm  Dept. 

3.3 

3.3 

3.3 

3.4 

3.3 

(14E) 

3.4 

3.5 

3.4 

3.4 

3.4 

Pat  Trn  Dept. 

4.4 

4.2 

4.4 

4.4 

4.4 

(24T) 

5.0 

5.0 

5.0 

5.0 

5.0 

Pat  Tm  Dept. 

5.0 

5.0 

5.0 

3.2 

4.6 

(Devices) 

5.0 

5.0 

5.0 

3.2 

4.6 

CATD 

3.0 

3.3 

3.0 

3.0 

3.1 

4.7 

4.7 

4.7 

4.3 

4.6 

32  AADCOM 

4.5 

4.8 

4.5 

4.5 

4.5 

OTT 

4.0 

4.5 

4.0 

4.0 

4.1 

32  AADCOM 

4.8 

4.9 

4.8 

4.8 

4.8 

Trn  &  Eval 

4.8 

5.0 

4.8 

4.2 

4.7 

Field  Unit  Personnel 

1  ITU  ADA  BDE 

2.5 

2.8 

2.5 

2.3 

2.5 

3.4 

3.4 

3.4 

3.2 

3.4 

6TH  ADA  BDE 

3.0 

3.0 

3.0 

2.3 

2.9 

3.7 

3.7 

3.7 

3.7 

3.7 

4/43  ADA 

5.0 

5.0 

5.0 

4.5 

4.9 

(32  AADCOM) 

5.0 

5.0 

5.0 

4.5 

4.9 

4/7  ADA 

3.5 

3.7 

3.5 

3.8 

3.4 

(32  AADCOM) 

4.3 

4.1 

4.3 

4.2 

4.2 

Hawk 

3.0 

3.0 

3.0 

2.7 

2.9 

Department 

3.0 

3.0 

3.0 

2.3 

2.5 

Note:  Timeliness  ratings  are  in  italics. 
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PAC  levels,  as  were  the  timeliness  ratings.  If  a  group  expected  to  use  MPMs  frequently, 
they  also  expected  to  use  DPHs  frequently.  Likewise,  if  a  group  required  access  to  DPHs 
only  quarterly,  they  required  access  to  MPMs  quarterly  as  well.  This  consistency  in  fre¬ 
quency  and  timeliness  ratings  seems  to  be  a  logical  extension  of  usefulness  ratings.  With 
the  usefulness  ratings,  groups  had  indicated  they  wanted  virtually  all  of  the  ARI  PAC 
measures.  With  the  frequency  and  timeliness  ratings  they  seem  to  be  saying  that  when 
they  want  one  measure,  they  want  them  all.  Indeed,  this  makes  some  sense.  For  a  given 
scenario,  it  is  difficult  to  predict  a  priori  which  levels  and  measures  will  be  needed  to  ade¬ 
quately  evaluate  and  understand  performance.  If  all  measures  are  always  available,  a 
thorough  evaluation  is  assured. 

The  variability  between  user  groups  in  frequency  and  timeliness  ratings  really  reflects 
how  individual  groups  go  about  their  training-related  activities.  Every  organization  seems 
to  be  different,  even  similar  types  of  groups.  For  example,  4/43  has  a  very  intense  training 
schedule  that  would  require  the  PAC  to  be  used  more  than  once  a  day  and  with  rapid  turn¬ 
around  of  feedback  (less  than  10  minutes  after  scenario  end).  Contrasted  with  4/43  is  the 
1 1th  BDE,  w'hich  anticipates  weekly  to  monthly  PAC  use  and  less  stringent  feedback  turn¬ 
around  requirements  (less  than  a  day).  For  the  organizations  that  deliver  training  (Patriot 
Department,  CATD,  32  AADCOM  Training  and  Evaluation  and  OTT  Sections),  a  range  of 
frequency  and  timeliness  values  are  also  observed.  However,  the  range  is  somewhat 
limited  and  the  values  tend  to  cluster  around  frequent  PAC  use  with  rapid  feedback  turn¬ 
around.  This  reflects  frequent  use  of  the  OTT  and  limited  time  in  the  training  environ¬ 
ment.  Finally,  the  lowest  ratings  of  frequency  and  timeliness  were  made  by  organizations 
that  analyze  training  effectiveness.  Ratings  provided  by  DOTD  and  DESCSD  indicate  an 
infrequent  need  for  PAC  data  (monthly  to  quarterly)  and  less  urgency  in  obtaining  results 
(greater  than  a  day  to  more  than  a  week).  Generally,  these  groups  formulate  specific  ques¬ 
tions  about  training  outcomes  and  need  a  sample  of  student  or  operator  data  to  answer 
those  questions.  The  sample  performance  data  can  be  accumulated  over  time.  When  a 
sufficient  sample  has  been  obtained,  the  analyst  can  then  export  the  PAC  data  to  another 
computer  for  study. 

The  last  ratings  provided  by  user  groups,  indicated  the  formats  in  which  they  wanted 
PAC  data  presented.  There  were  five  format  types  from  which  to  choose:  (1)  printouts, 

(2)  disk  or  tape  storage  and  export,  (3)  interactive  display  with  data  tables,  (4)  interactive 
display  with  data  presented  graphically,  (5)  expert  system  critique  with  replay.  Users 
could  select  more  than  one  format.  Results  were  similar  across  PAC  levels.  Table  4 
presents  results  of  format  selections  for  MPMs. 

As  with  the  ratings  of  usefulness,  timeliness,  and  frequency,  the  analysts  differed  on 
format  preferences  compared  to  the  trainers  and  field  unit  personnel.  The  analysts 
preferred  printouts  and  disk  or  tape  most  of  all,  consistent  with  their  use  of  accumulated 
data  for  review,  summarization,  and  analysis.  They  indicated  some  interest  in  interactive 
graphics  displays  and  expert  system  critique,  but  no  interest  in  interactive  tabular  displays  al 
all.  The  trainer  and  field  unit  personnel  ratings  were  more  evenly  spread  across  format 
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Table  4 


Percent  of  Each  Group  Requesting  the  Different  Categories  of  PAC  Feedback  Format  for 
the  MPMs 


Feedback  Format _ Groups 


Analysts 

Trainers 

Field  Unit 
Personnel 

Hawk 

Department 

Printouts 

83.3 

54.8 

44.4 

66.7 

Disk  or  Tape 

66.7 

9.7 

8.3 

33.3 

Interactive  Display 
(Data  Tables) 

0.0 

29.0 

11.1 

0.0 

Interactive  Display 
(Graphics) 

16.7 

35.5 

30.6 

66.6 

Expert  System  Critique 

16.7 

67.7 

55.6 

0.0 

with  Replay 

Note:  More  than  one  format  could  be  selected 


types.  They  were  most  interested  in  expert  system  critique  (i.e.,  "smart  scenario  replay"),  a 
potentially  powerful  teaching  tool.  Printouts,  which  provide  a  permanent  record  that  can 
be  used  as  a  refresher  or  reminder  after  scenario  end,  fell  a  close  second.  Interactive  dis¬ 
plays,  both  graphic  and  tabular,  a  completely  novel  format  for  the  Patriot  community,  fell 
third.  Trainers  and  field  unit  personnel  had  little  interest  in  the  use  of  disk  or  tape 
storage.  Personnel  from  the  Hawk  Department  favored  printouts  and  interactive  graphics 
displays  over  other  formats.  Of  greatest  interest,  however,  is  the  fact  that  none  of  them 
wanted  replay.  A  possible  explanation  is  that  Hawk  training  personnel  have  not  had  train¬ 
ing  simulators  in  the  past.  Consequently,  they  have  not  been  exposed  to  replay  and  its 
power  as  a  teaching  tool. 

In  addition  to  the  ratings,  the  user  groups  provided  comments.  A  complete  listing  of 
comments  obtained  is  provided  in  Appendix  E.  Comments  fell  into  three  categories:  ( 1 ) 
general  perceptions  of  the  feasibility  study  and  PAC  concept,  (2)  suggestions  for  new 
measures,  and  (3)  considerations  for  PAC  implementation  on  the  OTT  and  TPT.  With 
respect  to  general  perceptions  of  the  study  and  the  PAC  concept,  the  vast  majority  of  com¬ 
ments  were  very  favorable.  User  groups  were  delighted  to  have  the  opportunity  to  in¬ 
fluence  ti.t.  J^ign  of  a  tool  they  ultimately  would  be  given  to  use.  Also,  the  idea  of  an 
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automated  system  that  provides  an  objective  assessment  of  operator  performance  along 
with  an  in-depth  diagnostic  capability  was  generally  very  appealing.  As  a  whole,  the  user 
groups  recognized  that  lack  of  reliable,  accurate,  mission-based  operator  performance  infor¬ 
mation  is  a  major  obstacle  to  course  developers  and  trainers  providing  effective  training. 

The  PAC  was  viewed  enthusiastically  as  a  viable  solution  to  the  information  deficit. 

The  second  category  of  comments  consisted  of  suggestions  for  new'  measures.  Tw  o 
new  weapons  controller  FPMs  were  proposed.  The  first  is  average  To-Be-Engaged  Queue 
(TBEQ)  position  at  engage.  For  each  target  engaged  in  a  scenario,  this  measure  w'ould 
determine  its  rank  in  the  TBEQ  at  the  time  of  launch.  Ranks  would  be  averaged  across 
engagements  to  yield  the  final  score.  Scores  near  1.0  would  indicate  the  operator  tended 
to  shoot  out  of  the  top  of  the  queue.  This  measure  would  be  useful  for  scenarios  in  which 
firing  doctrine  dictates  that  the  queue  ordering  be  used.  The  second  measure  is  percent 
hostile  targets  killed  before  ordnance  release.  This  measure  would  examine  those  engage- 
able  hostile  aircraft  that  were  scripted  to  penetrate  assets.  It  would  assess  the  percent  that 
were  killed  before  they  could  reach  the  ordnance  delivery  point. 

The  third  category  of  comments  --  suggestions  and  considerations  for  PAC  implemen¬ 
tation  --  yielded  five  major  considerations.  They  were  all  practical  considerations  based 
on  an  intimate  understanding  of  the  environment  in  which  PAC  would  be  applied.  First, 
the  PAC  must  be  accurate  and  reliable  and  any  limitations  must  be  stated  explicitly.  Ac¬ 
curacy  and  reliability  are  key  factors  in  developing  and  maintaining  user  confidence  in  the 
system.  An  understanding  of  limitations  will  help  ensure  the  system  is  used  properly. 

How  misuse  could  occur  was  discussed  using  a  North  Atlantic  Treaty  Organization  (NATO) 
Tactical  Evaluation  as  an  example.  At  present,  exercise  controllers  can  change  system 
operating  states  such  as  method  of  control  at  will  during  an  evaluation.  The  PAC  depends 
heavily  upon  pre-defined  identification  and  engagement  windows  to  evaluate  performance. 
In  order  to  define  these  windows,  the  scenario  script  (including  system  operating  states) 
must  be  known.  If,  during  an  evaluation,  the  controller  deviates  from  the  script,  the  result¬ 
ing  PAC  evaluation  might  be  in  error.  Evaluators  must  be  aware  of  this  limitation  and  its 
effect  on  the  meaningfulness  of  the  PAC  measures. 

On  the  other  hand,  the  PAC  must  be  able  to  reflect  local  TSOP  in  its  assessment. 

When  a  training  scenario  is  developed,  the  designer  uses  a  selected  TSOP.  That  scenario 
is  then  sent  to  units  and  training  organizations  that  might  use  a  different  TSOP.  The 
TSOP  is  one  of  the  factors  that  drives  specification  of  PAC  identification  and  engagement 
windows.  If  the  TSOP  changes,  the  windows  can  change.  Therefore,  some  capability 
must  be  provided  so  that  units  and  training  organizations  can  revise  PAC  windows  to  reflect 
their  TSOP  prior  to  running  a  scenario.  Whatever  system  is  devised,  it  must  be  easy  to  use 
and  require  little  time  to  make  the  necessary  changes. 

Critically,  the  Patriot  enlisted  operator  trainers  stressed  the  need  for  a  PAC  that  was 
built  around  their  Program  of  Instruction  (POI)  and  that  would  automate  test  and  scoring 
currently  done  manually  by  instructors.  This  group’s  ratings  of  the  ARI  PAC  were 
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consistently  lower  than  those  of  the  other  groups,  in  large  part  because  the  measures  do  not 
directly  reflect  the  performance  evaluation  criteria  used  in  their  POl.  These  users  sug¬ 
gested  that  tests  administered  in  existing  training  programs  be  reviewed  to  identify  perfor¬ 
mance  measures  used  and  that  they  be  added  to  the  PAC. 

Also,  users  emphasized  the  need  to  provide  an  effective  replay  capability.  A  critical 
feature  of  effective  replay  is  the  ability  to  quickly  access  a  particular  time  period  in  a 
scenario  and  start  replay  at  that  point.  The  current  OTT  and  TPT  provide  replay  but  lack 
the  ability  to  access  time  periods  quickly.  Valuable  training  time  is  lost  waiting  for  the 
simulator  to  reach  a  time  period  of  interest.  Consequently,  instructors  don’t  use  replay  as 
often  as  they  would  like. 

Finally,  the  Patriot  community  should  expect  the  PAC  to  change  with  use.  As  with 
any  new  system,  areas  for  improvement  and  new  applications  will  be  discovered  once  the 
system  is  implemented.  A  PAC  support  network  is  required  in  which  feedback  from  users 
is  obtained,  system  modifications  are  made,  and  software  changes  and  new  information  is 
disseminated  back  to  the  user  community. 

Results  of  sub-task  1.2;  Identify  additional  task  measures.  Table  5  lists  the  14E  and 
24T  tasks  that  were  selected  by  the  working  group  for  assessment  by  an  OTT  and  TPT  PAC. 
The  ta^ks  fall  into  two  areas:  system  initialization  and  compulsory  safety  procedures.  With 
the  exception  of  the  engagement  tasks,  all  of  the  tasks  in  the  list  are  new  tasks  not  ad¬ 
dressed  by  the  ARI  PAC.  The  ARI  PAC  does  assess  target  engagement,  but  does  not  dif¬ 
ferentiate  between  different  types  of  engagements  such  as  engagement  of  jammers  and 
tactical  ballistic  missiles  (TBMs).  Developing  measures  for  assessing  these  tasks  was  out 
of  the  scope  of  this  study;  however,  the  POIs  used  to  train  these  tasks  should  be  a  good 
source  of  information  for  measure  development. 

Results  of  Task  2:  Specify  Data  Collection  Requirements 

The  primary  product  of  Task  2  was  an  ARI  PAC  performance-measure-by-data-re- 
quired-element  matrix.  The  complete  matrix  is  presented  in  Appendix  F.  Figure  3 
presents  an  excerpt.  Listed  in  the  rows  of  the  matrix  are  the  ARI  PAC  measures.  (For¬ 
mulae  for  the  measures  are  presented  in  Appendix  G.)  In  the  columns  are  the  data  ele¬ 
ments  used  to  calculate  PAC  measures.  Essentially,  the  matrix  specifies  the  data  elements 
that  must  be  collected  from  a  Patriot  simulator  to  support  computation  of  the  PAC 
measures  and  indicates  which  data  elements  feed  which  measures. 

Results  of  the  analysis  of  OTT  data  collection  are  presented  in  Figure  4.  The  matrix 
relates  PAC  data  elements  to  variables  and  data  messages  available  in  the  OTT.  Only 
those  data  elements  that  are  collected  by  the  PAC  during  a  scenario  run  are  listed.  All 
other  data  elements  (see  Appendix  F)  are  generated  by  the  PAC  itself.  The  matrix 
demonstrates  that  all  of  the  data  needed  to  calculate  PAC  measures  are  available  in  the 
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Table  5 


14E  and  24T  Soldier  Tasks  Recommended  for  Use  in  an  OTT  and  TPT  PAC 


Task  Number 

01-0401.05-TBD 

01-0401.05-0046 

01-0401.05-0047 

01-0401.05-0048 

01-0401.05-0049 

01-0401.05-0089 

01-0401.05-0090 

01-0401.05-0091 

01-0401.05.0092 

01-0401.05-0093 

01-0401.05-0094 

01-0401.05-TBD 

01-0401.05-0417 

01-0401.05-0418 

01-0401.05-0149 

01-0401.05-0150 


Task  Number 
441-083-1407 

441-083-1124 

441-083-1409 

441-084-1125 

441-083-1471 

441-083-1472 

441-083-1473 

441-083-1474 

441-083-1475 

441-083-1476 

441-083-1477 

441-083-1478 

441-083-1479 

441-083-1480 

441-083-1481 

441-083-1482 

441-083-1486 

441-083-1487 

441-083-1488 

441-083-1490 

441-083-1491 

441-083-1492 

441-084-1114 


14E  Tasks 


Task 


Energize  the  Information  Coordination  Central 

Supervise  manual  tactical  software  initialization  in  the  Information  Coordination 
Central  (ICC) 

Supervise  automatic  tactical  software  initialization  in  the  Information  Coordination 
Central  (ICC) 

Supervise  initialization  of  software  using  last  prior  data  base  (LPDB) 

Supervise  recovery  operations  in  the  Information  Coordination  Central  (ICC) 

Perform  protection  of  friendly  aircraft  entering  the  Battalion  Area  of  Responsibility  in 
the  Information  Coordination  Central  (ICC) 

Perform  engagement  of  targets  from  the  Information  Coordination  Central  (ICC) 
Monitor  tactical  situations  and  status  of  battalion  response  to  tactical  requirements 
Perform  alternate  deployment  activation  in  the  Information  Coordination  Central 
(ICC) 

Perform  a  fire  platoon  initialization  support  request  in  the  Information  Coordination 
Central  (ICC) 

Perform  fire  platoon  data  base  comparison  in  the  Information  Coordination  Central 
(ICC) 

Send  free  form  message  from  the  Information  Coordination  Central  (ICC) 

Supervise  the  Firing  Battery  Air  Defense  Battle  in  Centralized  Mode 
Supervise  the  Firing  Battery  Air  Defense  Battle  in  Decentralized  Mode 
Supervise  the  Firing  Battery  Air  Defense  in  CentralizedMode 
Supervise  the  Firing  Battery  Air  Defense  in  Autonomous  Mode 

24T  Tasks 

Task _ 

Perform  as  Crew  Member  No.l  (MSl)during  Engagement  Control  Station  (ECS) 
initialization 

Perform  as  Crew  Member  No.3  (MS3)  during  Engagement  Control  Station  (ECS) 
initialization 

Perform  as  Crew  Member  No.l  (MSI)  during  Information  and  Coordination 
Central  (ICC)inilialization 

Perform  as  Crew  Member  No.3  (MS3)  during  Information  and  Coordination  Central 
(ICC)  initialization 
Activate  fire  unit 

Activate  Information  and  Coordination  Central  (ICC) 

Change  configuration  from  on-line  to  primary/secondary  network  Information  and 
Coordination  Central  (ICC) 

Deactivate  fire  unit  Engagement  Control  Station  (ECS) 

Deactivate  Information  and  Coordination  Central  (ICC) 

Engage  jammers-Engagement  Control  Station  (ECS) 

Engage  tactical  ballistic  missile-Engagement  Control  Station  (ECS) 

Engage  targets-Engagement  Control  Station  (ECS) 

Evaluate  pre-engagement  data-Engagement  Control  Station  (ECS) 

Evaluate  pre-engagement  data-Information  and  Coordination  Central  (ICC) 

Initiate  jammer  engagements-Information  and  Coordination  Central  (ICC) 

Initiate  target  engagements-Information  and  Coordination  Central  (ICC) 

Perform  friendly  proteci-Engagcment  Control  Station  (ECS) 

Perform  friendly  protect  Information  Coordination  Central  (ICC) 

Perform  missile  hazard/misfire  procedures 

Perform  reinitialization  in  the  Engagement  Control  Station  (ECS) 

Perform  saturation  alleviation  proccdures-Engagcmcnt  Control  Station  (ECS) 

Perform  system  reorientation  and  clutter  map  update  (CMUP)-  Engagement 
Control  Station  (ECS) 

Perform  compulsory  safety  procedures 
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PAC  MEASURES 

attrition 

friendly  protection 
%  hostiles  engaged 
9c  hostiles  killed 
etc. 


REQUIRED  DATA  ELEMENTS 
flight  number 

scripted  ID 


i  flight  start  size 

i 


Figure  3.  Excerpt  from  ARI  PAC  measure  by  data  element  matrix. 

OTT.  Thus,  in  terms  of  data  availability,  it  is  possible  to  implement  a  PAC  on  the  current 

or r. 

Results  of  Task  3:  Specify  and  Evaluate  PAC  Implementation  Schemes 

The  results  of  Task  3  are  discussed  in  terms  of  the  three  main  products  of  the  task:  ( 1 ) 
a  PAC  operational  concept.  (2)  implementation  alternatives  for  the  current  OTT,  and  (3)  a 
sample  user  interface. 

A  PAC  operational  concept.  The  results  of  Task  1  enabled  specification  of  some 
basic  requirements  for  an  OTT  and  TPT  PAC.  The  PAC  must  (1)  compute  all  of  the  ARI 
PAC  performance  measures,  (2)  support  assessment  of  additional  system  initialization  and 
compulsory  safety  procedures,  (3)  support  schoolhouse  testing,  (4)  provide  feedback  within 
ten  minutes  after  scenario  end,  (5)  archive  performance  data,  and  (6)  provide  a  replay 
capability  that  quickly  accesses  a  specific  time  period  in  the  scenario.  Also,  given  the 
ability  of  the  OTT  to  run  multiple  students  on  different  scenarios  simultaneously,  a  seventh 
requirement  was  added.  (7)  The  PAC  must  be  able  to  evaluate  up  to  eight  operators 
simultaneously  and  still  provide  feedback  within  ten  minutes  after  scenario  end. 

A  PAC  operational  concept  was  developed  that  meets  the  requirements  specified 
above.  As  shown  in  Figure  5,  the  PAC  is  linked  in  some  fashion  to  the  OTT  or  TPT 
simulation.  As  the  simulation  runs,  occurrences  of  critical  events  (e.g.,  target  identifica¬ 
tions  and  engagements,  operator  switch  actions)  are  trapped.  Data  collection  messages 
that  contain  critical  event  information  are  generated  and  sent  to  the  PAC  where  they  are 
captured  and  processed  by  the  data  collection  and  summarization  module.  The  function 
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Figure  4.  Cross-reference  of  ARI  PAC  data  elements  and  OTT  data  collection 

of  this  module  is  to  capture  data  sent  by  the  simulation,  interpret  it,  and  use  it  to  update  a 
data  summary  table  in  realtime. 

The  data  summary  table  is  a  matrix  that  resides  in  memory  in  the  PAC  host  computer. 
Columns  of  the  matrix  are  made  up  of  the  PAC  data  elements  specified  in  Task  2.  In¬ 
cluded  in  these  data  elements  are  predefined  variables  such  as  flight  group  number, 
scripted  identification,  and  window  start  and  end  times  (refer  again  to  Appendix  F).  Rows 
of  the  matrix  correspond  to  flights  in  the  scenario.  Thus,  the  data  summary  table  provides 
a  means  for  summarizing  the  identification,  engagement,  and  other  supporting  actions 
(hooks,  IFF,  etc.)  taken  on  each  flight  in  a  scenario.  At  the  end  of  a  scenario,  data  in  the 
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PERFORMANCE  MEASURES 

Mission  Performance  Measures:  defense  of  assets,  attrition,  etc. 

Function  Performance  Measures  91  hostiles  killed,  9F  engagements  late,  %  tracks  IDed  early,  etc 
Task  Performance  Measures.  °!c  tracks  hooked,  %  alerts  displayed,  etc. 

Detailed  Performance  Histones 


Figure  5.  OTT  and  TPT  PAC  operational  concept. 
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summary  table  are  used  to  compute  PAC  performance  measures.  Development  of  the 
summary  table  as  the  simulation  runs  permits  measure  calculation  to  be  performed  immedi¬ 
ately  after  scenario  end.  Consequently,  feedback  presentation  delay  is  minimized. 

Once  PAC  measures  are  generated,  the  summary  table  data  and  the  measures  are 
written  to  files  for  archiving  purposes.  Now  the  measures  and  data  are  available  for  view¬ 
ing.  A  user  feedback  interface  controls  presentation  of  PAC  data.  With  the  feedback  in¬ 
terface,  a  user  can  selectively  view  data  from  the  different  PAC  levels,  diagnose  classes  of 
performance  problems,  and  isolate  specific  instances  of  problems.  Ideally,  the  scenario 
replay  feature  of  the  simulation  is  linked  to  the  PAC  feedback  interface.  Once  problem 
points  in  a  scenario  are  identified,  the  replay  feature  can  access  those  points  and  critique 
student  actions.  It  is  from  the  user  interface  that  printouts  and  reports  of  PAC  data  are  in¬ 
itiated. 

Finally,  the  PAC  measure  formulae  and  data  element  information  presented  in  Ap¬ 
pendices  F  an  G  are  based  on  the  ARI  PAC  that  is  oriented  toward  assessment  of  in¬ 
dividual  operators  and  friendly  protector  and  weapons  con 'roller  crews.  As  such,  it 
provides  a  substantial  basis  for  assessment  of  collective  performance.  The  notion  of  assess¬ 
ing  decisions  made  within  time  windows  is  still  applicable;  however,  the  process  for  specify¬ 
ing  windows  must  change  to  account  for  the  fact  that  multiple  fire  units  can  track  and 
process  the  same  aircraft.  For  example,  specifying  the  v-,,->ctive"  engagement  window  for 
a  scripted  hostile  must  consider  that  more  than  one  "'re  anil  will  have  the  opportunity  to 
engage  an  aircraft  before  it  can  penetrate  .n  asset.  The  window  start  for  the  track  would 
be  the  first  point  in  time  that  the  first  tire  unit  able  to  engage  the  aircraft  can  do  so  with  a 
high  Pk.  The  window  end  point  would  be  th.r  last  point  in  time  that  the  last  fire  unit  able 
to  engage  the  aircraft  can  engage  and  intercept  betore  it  penetrates  the  asset. 

The  assessment  of  decision  making  in  the  collective  environment  must  also  change  to 
accommodate  the  increased  number  of  personnel  involved.  The  same  basic  measures 
(MPMs,  FPMs,  and  TPMs)  used  to  assess  individual  and  crew  decision  making  can  still  be 
used  but  the  diagnostic  process  is  different.  Suppose,  for  example,  that  a  value  of  25%  is 
observed  for  the  FPM  percent  hostiles  engaged  late  after  a  collective  training  scenario. 

The  diagnostic  process  would  involve  reviewing  PAC  engagement  data  for  each  of  the 
aircraft  engaged  late  and  evaluating  the  decision  process  to  see  where  decision  making  was 
delayed.  Was  the  weapons  controller  at  the  battalion  Information  Coordination  Central 
late  in  initiating  the  engagement?  Or  was  the  weapons  controller  at  the  fire  unit  Engage¬ 
ment  Control  Station  late  in  processing  alerts  or  in  responding  to  engagement  alerts?  In 
summary,  the  ARI  PAC  provides  a  good  basis  for  evaluating  both  individual  and  collective 
performance.  Further  specification  is  required,  however,  to  permit  definition  of  the 
decision  windows  and  the  effective  diagnosis  of  collective  performance. 

Implementation  alternatives  for  the  current  OTT.  Results  of  Task  2  demonstrated 
that  the  data  required  to  compute  PAC  measures  are  currently  available  in  the  OTT.  In 
this  portion  of  Task  3,  two  alternatives  for  implementing  the  PAC  on  the  existing  OTT  were 
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ALTERNATIVE  1 


ALTERNATIVE  2 


EVALUATION 

FACTORS: 
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Figure  6.  Comparison  of  alternatives  for  implementing  PAC  on  current  OTT. 

specified  and  evaluated.  The  first  alternative  retains  the  current  computers  and  requires 
that  the  current  data  collection  and  reduction  software  be  reworked  to  incorporate  the 
necessary  PAC  functions.  The  second  alternative  requires  that  the  PAC  be  hosted  on  its 
own  computer.  Within  this  concept,  the  PAC  is  linked  to  the  OTT  via  the  two  buses  used 
to  pass  scenario  data  message  traffic  in  the  simulation.  This  provides  the  data  collection 
portion  of  the  PAC  access  to  the  simulation  event  data  required  by  the  PAC.  All  PAC 
processing  is  allocated  to  the  PAC  computer.  There  is  no  requirement  to  rework  OTT 
software. 

In  evaluating  the  two  implementation  alternatives,  five  factors  were  considered:  (1)  es¬ 
timated  relative  software  costs,  (2)  estimated  hardware  costs,  (3)  potential  impact  on  OTT 
simulation  reliability,  (4)  potential  impact  on  OTT  realtime  performance,  and  (5)  com¬ 
patibility  with  future  system  upgrades.  Figure  6  presents  the  results  of  the  comparison  of 
the  two  alternatives.  Estimated  software  costs  for  the  first  alternative  are  expected  to  be 
higher  than  those  for  the  second  alternative.  Both  alternatives  require  that  the  same  basic 
functions  be  implemented;  however,  the  first  alternative  will  also  require  an  extensive 
analysis  of  the  current  evaluation  software  to  determine  which  portions  to  keep  and  which 
to  replace  with  PAC  software.  Hardware  costs,  on  the  other  hand,  would  be  considerably 
higher  for  the  second  alternative:  Although  additional  memory  might  be  required  under 
the  first  alternative,  the  cost  of  that  memory  would  be  considerably  less  than  the  separate 
PAC  computer  required  for  each  OTT  suite  under  the  second  alternative. 
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The  first  alternative  is  likely  to  negatively  affect  both  simulation  reliability  and  real¬ 
time  performance.  It  would  require  a  major  modification  of  the  existing  software  and 
there  is  always  a  risk  associated  with  such  a  retrofit.  Further,  with  the  addition  of  the  PAC 
software,  it  would  place  a  significant  computing  burden  on  the  OTT  computer,  jeopardizing 
any  realtime  capability,  especially  with  large  scenarios.  Conversely,  the  second  alternative 
would  require  no  modification  of  the  existing  OTT  software  and  all  PAC  processing  would 
be  on  the  PAC  computer.  It  is  even  possible  that  the  second  alternative  may  permit  the 
deactivation  of  certain  OTT  data  collection  activities  thereby  enhancing  the  OTT  realtime 
capability. 

Finally,  the  OTT  is  planned  to  be  upgraded  in  the  next  few  years  to  provide  a  simula¬ 
tion  of  the  Patriot  Post-Deployment  Build  Three  (PDB-3)  software  and  increase  the 
system’s  computing  power.  To  be  most  cost  effective,  any  performance  evaluation  im¬ 
provements  made  to  the  current  OTT  should  transfer  to  the  upgrade  with  little  modifica¬ 
tion.  If  the  current  OTT  software  is  to  be  used  as  the  basis  for  the  PDB-3  update, 
alternative  one  would  transfer  smoothly.  However,  alternative  one  would  require  a  consid¬ 
erable  effort  to  develop  a  PAC  if  completely  new  software  were  procured,  although  it 
would  have  the  advantage  of  being  developed  from  the  ground  up  and  integrated  fully  with 
the  simulation.  On  the  other  hand,  alternative  two  should  transfer  quite  easily  to  an 
upgrade.  The  only  portion  that  might  require  modification  is  the  data  collection  module. 
This  would  be  required  if  the  type  and  content  of  the  data  collection  messages  in  the  PDB-3 
simulation  were  different  from  the  current  simulation. 

All  factors  considered,  alternative  two  is  recommended.  Tnough  it  is  probably  a 
more  expensive  hardware  investment  than  alternative  one,  it  is  less  likely  to  adversely  affe  n 
functioning  of  the  current  OTT  and  more  likely  to  transfer  smoothly  to  an  OTT  upgrade  sys¬ 
tem. 


Note  that  the  PAC  operational  concept  described  previously  applies  to  both  the  OTT 
and  TPT  environments  but  that  implementation  is  discussed  for  only  the  OTT.  Limita¬ 
tions  may  well  exist  for  PAC  implementation  on  the  TPT.  The  TPT  runs  on  the  tactical 
system  Weapons  Control  Computer  (WCC)  which  is  limited  in  the  amount  of  memory  and 
processing  power  available  for  PAC  operations.  PAC  implementation  on  the  TPT  would 
require  an  analysis  of  TPT  operation  on  the  WCC  and  an  evaluation  of  PAC  memory  and 
processing  requirements  to  determine  whether  all  or  just  portions  of  the  PAC  can  be  imple¬ 
mented  in  the  TPT  without  degrading  realtime  performance.  Such  analysis  and  evaluation 
were  beyond  the  scope  of  this  study. 

Sample  user  interface  screens.  The  sample  user  interface  screens  w’ere  developed  to 
provide  a  tangible  point  of  departure  for  discussing  a  PAC  user  interface.  They  were  not 
considered  to  be  a  final  recommendation  for  a  PAC  user  interface.  As  part  of  implement¬ 
ing  the  PAC  on  an  OTT  and  TPT,  an  in-depth  study  of  the  user  interface  will  be  required. 
This  «fudv  should  carefully  adhere  to  established  human  factors  principles  in  computer  in¬ 
terface  design  and  should  involve  user  groups  extensively. 
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Figure  7.  Sample  PAC  user  interface  screen. 

In  the  sample  screens,  a  diagnostic  sequence  is  provided  in  which  a  user  can  select  an 
operator  and  determine  factors  contributing  to  poor  friendly  protector  performance.  In 
the  diagnostic  sequence,  the  user  first  selects  an  operator  and  a  scenario.  Mission  perfor¬ 
mance  measures  (MPMs)  are  displayed  for  that  scenario.  The  user  can  then  position  a 
light  bar,  or  highlighted  cursor,  over  "Friendly  Protection  MPM"  and  press  ENTER  to  ob¬ 
tain  a  display  of  relevant  friendly  protector  FPMs.  From  the  FPMs,  the  user  can  use  the 
light  bar  to  select  "Percent  Friends  Identified  Late."  The  system  then  displays  a  list  of 
friendly  aircraft  identified  late  along  with  critical  data  such  as  identification  window  end 
and  identification  time.  For  each  one  of  the  aircraft,  a  detailed  history  can  be  obtained  by 
positioning  the  light  bar  over  the  aircraft  data  and  pressing  ENTER.  Thus,  the  sample 
user  interface  is  characterized  by  two  primary  features:  (1)  a  point-and-shoot  interaction 
in  which  the  user  directs  the  system  by  light  bar  and  single  key  commands,  all  prompted  by 
the  display  screens,  and  (2)  a  pre-defined  organization  of  data  in  which  selection  of  data  at 
one  level  results  in  presentation  of  specific  data  elements  in  the  next  level.  Figure  7 
presents  a  sample  user  interface  screen.  The  figure  illustrates  the  screen  that  would  be  dis¬ 
played  if  a  user  selected  scenario  1 1  for  student  number  26  and  then  requested  FPMs  re¬ 
lated  to  mission  performance.  At  the  top  of  the  screen,  header  data  indicate  the  student 
and  scenario  selected  and  provide  background  information  on  the  scenario  such  as  average 
window  size  (S  =  small),  average  DWE  or  delta  window  end  (B  -  big),  and  number  of 
tracks  in  the  scenario  (L  =  low).  The  next  row  of  boxes  present  MPMs.  The  last  row  of 
boxes  present  FPMs  associated  with  the  friendly  protector  MPM. 

The  sample  user  interface  screens  were  demonstrated  to  two  user  groups,  the  instruc¬ 
tors  for  the  14E  (Patriot  officer)  training  course  and  the  instructors  for  the  24T  (Patriot  en- 
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listed)  training  course.  The  14E  trainers  were  very  positive  in  their  review  of  the  screens. 
They  felt  the  point-and-shoot  interface  made  the  system  easy  to  use.  Also,  they  liked  the 
ability  to  trace  performance  problems  through  multiple  PAC  levels  and  "see"  the  different 
levels.  Finally,  they  had  no  problem  with  use  of  a  pre-defined  organization  of  the  data. 
They  felt  it  simplified  use  of  the  system. 

As  with  the  ARI  PAC,  the  24T  trainers  were  negative  in  their  response  to  the  sample 
user  interface  screens.  While  they  agreed  that  the  format  was  easy  to  use,  they  had  two 
major  criticisms.  (1)  The  measures  provided  by  the  system  must  be  consistent  with  the 
standards  and  criteria  provided  in  the  POI.  (2)  The  instructor  has  a  very  limited  amount 
of  time  for  providing  feedback.  While  the  feedback  provided  by  the  system  is  precise  and 
in-depth,  there  is  concern  that  it  might  take  too  long  to  administer  and,  as  a  consequence, 
increase  course  completion  times.  A  possible  compromise  might  be  to  use  detailed  PAC 
feedback  selectively,  only  at  critical  points  in  the  POI. 

Recommendations 

In  summary,  the  PAC  feasibility  study  objective  was  to  determine  the  feasibility  of 
using  the  ARI-developed  PAC  as  the  basis  for  an  improved  operator  performance  evalua¬ 
tion  system  on  Patriot  trainers.  The  study  was  initiated  through  extensive  meetings  with 
representatives  of  virtually  all  Patriot  trainer  user  groups.  Ratings  were  obtained  on  the 
usefulness,  the  required  frequency  and  timeliness,  and  the  desired  format  of  the  ARI  PAC 
measures.  In  addition,  a  review  of  soldier  task  lists,  analysis  of  OTT  data  collection 
software,  and  evaluation  of  various  PAC  implementation  alternatives  were  conducted.  Al¬ 
though  the  emphasis  at  the  inception  of  the  study  was  on  the  implementation  of  a  PAC  on 
the  existing  OTT,  it  immediately  shifted  to  the  specification  of  a  functional  PAC  concept. 
Thus,  based  on  the  information  gathered,  the  following  suggested  recommendations  are 
made  for  a  Patriot  training  PAC,  all  of  which  must  be  integrated  with  other  technical  re¬ 
quirements  put  forth  by  DOT'D: 

1.  Include  all  measures  and  data  from  the  ARI  PAC. 

2.  Add  measures  derived  from  soldier  task  lists  related  to  system  initialization  and 
system  operational  procedures,  and  refine  ARI  PAC  target  engagement  measures  (e.g., 
engagement  of  jammers,  TBMs,  and  non-jamming  air  breathing  threats). 

3.  Tie  PAC  to  POI  for  schoolhouse  use. 

4.  Provide  PAC  feedback  within  ten  minutes  after  scenario  end. 

5.  Provide  data  archiving  to  support  analysts. 

6.  Provide  flexible  replay  as  an  essential  PAC  feature. 
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7.  Provide  capability  to  assess  multiple  operators  running  multiple  simultaneous 
scenarios. 

8.  Extend  PAC  to  assess  collective  performance. 

9.  Provide  a  flexible,  easy  to  use  means  of  modifying  PAC  identification  and  engage¬ 
ment  windows  so  that  users’  local  TSOP  can  be  reflected  in  the  PAC  evaluation. 

10.  Design  feedback  interface  to  maximize  ease  of  use.  Continue  user  input 
throughout  design  and  implementation  process. 

Two  final  words  are  due  about  the  implications  of  a  PAC  implementation:  standards 
and  scenario  difficulty  scaling.  Throughout  this  report,  the  notion  of  assessment  has  been 
central  to  the  discussion  of  the  PAC.  This  particular  performance  assessment  capability 
has  been  described  as  descriptive,  diagnostic,  and  decision-driven.  This  PAC  has  not,  how¬ 
ever,  been  described  as  an  evaluation  system.  Implied  in  the  concept  of  evaluation  is  the 
availability  of  standards  against  which  performance  data  are  compared.  This  PAC  does 
not  carry  standards  with  it,  but  it  does  support  the  research  required  to  develop  and  estab¬ 
lish  performance  standards  that  can  be  tailored  to  the  training  goal,  the  unit  TSOP,  and  tac¬ 
tical  realism.  By  the  same  token,  the  concept  of  the  PAC  windows  serves  as  the  basis  for 
scaling  scenarios  according  to  the  varying  difficulty  associated  with  the  training  goal,  the 
unit  TSOP,  and  tactical  realism. 
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APPENDIX  A 


Descriptions  and  Definitions  of  ARI  PAC  Measures 
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MPM  DEFINITIONS 
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FUNCTION  PERFORMANCE  MEASURES 
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FRIENDLY  PROTECTOR  FUNCTION 
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DEFINITIONS  OF  BASIC  FRIENDLY 


DEFINITIONS  OF  FRIENDLY  PROTECTOR  FPM 
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APPENDIX  B 


User  Group  Rating  Form 
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INSTRUCTIONS  FOR  RATING  PCOFT/TPT  PAC 
CANDIDATE  PERFORMANCE  MEASURES 


Future  PCOFT/TPT  PAC  user: 

This  is  your  opportunity  to  influence  development  of  the  PCOFTTPT  PAC.  You  have  been  given  jn  over¬ 
view  of  the  PTOS  PAC  which  described  its  structure  and  levels,  defined  the  measures,  and  described  how  they 
are  used.  Now  you  are  asked  to  rate  each  measure  to  indicate  how  useful  they  would  be  to  you  in  your  job 
The  ratings  you  provide  will  be  used  to  select  of  the  measures  to  be  included  in  the  PCOFT/TPT  PAC.  Please 
follow  the  instructions  provided  below  and  make  the  ratings  to  the  best  of  your  abilitv.  Thank  you. 


Organization  of  the  Form 

There  are  three  primary  sections  to  the  rating  form.  In  the  first  section  you  will  indicate  the  different  wavs 
you  might  use  PAC  data.  In  the  second  section  you  will  rate  the  performance  measures  themselves.  And 
finally,  any  comments  you  have  about  the  PAC  can  be  made  in  section  three. 


Rating  Dimensions 

Before  starting  the  rating  process,  we  want  to  describe  the  four  dimensions  or  factors  used  for  ratine  the 
candidate  performance  measures.  Each  dimension  is  listed  and  described  on  the  next  page.  The  firsi  dimen¬ 
sion.  usefulness,  will  be  used  to  rate  the  candidate  performance  measures  individually.  The  remaining  three 
will  be  used  to  rate  the  candidate  performance  measures  by  groups  within  the  different  levels  of  the  PTOS 
PAC.  Please  read  the  descriptions  carefully  and  make  the  ratings  as  accurately  as  possible.  If  you  have  anv 
questions,  please  don’t  hesitate  to  ask. 


1 
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DESCRIPTION  OF  RATING  DIMENSIONS 


Usefulness: 

How  helpful  the  measure  is  to  you  in  your  work.  Assesses  how  well  it  might  answer  questions  you  would 
ask  about  operator  performance. 


1  2  3  4  5 

not  useful  somewhat  useful  very  useful 


Frequency: 

An  estimate  of  how  often  you  might  need  a  measure. 


1  2  3 

quarterly  or  monthly  weekly 

less 


4  5 

once  a  day  several  times  a  dav 


Timeliness: 

An  estimate  of  how  quickly  you  might  need  a  measure  after  a  scenario  run  or  set  of  scenario  runs. 


1  2  3  4  5 

week  or  1  week  a  day  1  hour  10  min.  or  less 

longer 


Format: 

A  description  of  the  format  in  which  the  data  are  needed. 


1 

printout 


3 

4 

> 

disk,  tape 

interactive 

interactive 

expert  svstem 

display 

display 

critique  w  replay 

(data  tables) 

(graphics) 
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RATING  FORM  FOR  PCOFT/TPT  CANDIDATE  PERFORMANCE 

MEASURES 


Date: 


Organization: _ 

Job  Title/Ranle _ 

Indicate  How  You  Will  Use  the  PAC 

The  PCOFT/TPT  P.AC  will  serve  a  number  of  different  user  groups.  You  have  been  selected  as  a  repre¬ 
sentative  of  a  particular  group.  Think  of  the  types  of  activities  you  and  others  in  your  group  perform  and  how 
informatioa  data  from  a  PAC  could  be  used  to  support  those  activities.  From  the  list  below,  circle  the  item(s) 
that  best  describe(s)  how  you  would  use  PAC  data. 

Circle  items  in  this  column 

Performance  Scoring  End  of  course/end  of  module  assessment 

operator  certification/ qualification 
MOS  qualification 
standards  development 


Performance  Diagnostics  instructor  curriculum  choices 

courseware  evaluation/ modification 
training  requiremenls/shortfalls 

Unit  Readiness  mission  training  plans 

ARTEPS 

unit  readiness  reports 


Training  Administration/Tracking  training  eflectivenesss  analyses 

end  of  course  proficiencies 
learning'course  completion  rates 
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RATING  FORM  FOR  PCOFT/TPT  CANDIDATE 
MISSION  PERFORMANCE  MEASURES 

Rating  the  Mission  Performance  Measures 

In  this  section,  consider  the  PTOS  PAC  mission  performance  measures.  First,  rate  'r.f„4  mr  provided  by 
each  measure  on  the  dimension  of  usefulness.  Next,  rate  the  mission  performance  .*.i_^ouies  as  a  group  in 
terms  of  frequency,  timeliness,  and  format.  To  rate  an  item  on  a  dimension,  circle  the  value  of  the  rating  vcv 
wish  to  assign  on  the  scale  across  from  the  item  being  rated.  As  you  make  the  ratings,  remember  to  rate  each 
item  as  it  relates  to  your  information/data  needs.  Feel  free  to  refer  to  the  descriptions  of  the  racing  dimensions 
and  to  the  briefing  slides  to  refresh  your  memory  of  the  mission  performance  mc^.m-s. 


Rate  the  Usefulness  of  the  Mission  Performance  Measures 

Consider  the  mission  performance  measures  listed  below.  Rate  each  one  in  terms  of  how  useful  it  can  be  to 
you  in  your  job. 


not  useful 

Somewhat 

> erv  usefui 

Measure 

useful 

defense  of  assets 

i  : 

3 

* 

5 

attrition 

i  : 

3 

4 

5 

friendly  protection 

t  : 

3 

4 

5 

resource  conservation 

t  : 

3 

4 

5 

Rate  the  Frequency,  Timeliness,  and  Format  of  the  Mission  Performance  Measures 

Now,  consider  the  mission  performance  measures  as  a  group.  Rate  them  in  terms  of  the  frequency,  timeli¬ 
ness,  and  format  in  which  you  would  need  them. 


Frequency: 

t 

quarterly  or 
less 

monthly 

3 

weekly 

4 

once  a  day 

5 

several  times  a  day 

Timeliness: 

1 

3 

4 

5 

*eek  or 
longer 

l  week 

a  day 

l  hour 

10  min.  or  less 

Format: 

t 

3 

4 

5 

printout 

disk,  tape 

interactive 

interactive 

expert  vysteni 

display 

display 

critique  *  repljy 

(dala  tablesi 

(graphics) 
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RATING  FORM  FOR  PCOFT/TPT  FRIENDLY  PROTECTOR 
CANDIDATE  FUNCTION  PERFORMANCE  MEASURES 


Rating  the  Friendly  Protector  Function  Performance  Measures 
In  this  section,  consider  the  PTOS  PAC  Friendly  Protector  function  performance  measures.  As  before,  rate 
information  provided  by  each  measure  on  the  dimension  of  usefulness  first.  Then,  rate  the  Friendly  Protector 
function  performance  measures  as  a  group  in  terms  of  frequency,  timeliness,  and  format.  Remember  to  rate 
each  item  as  it  relates  to  your  information/data  needs.  Feel  free  to  refer  to  the  descriptions  of  the  rating 
dimensions  and  to  the  briefing  slides  to  refresh  your  memory  of  the  function  performance  measures. 


Rate  the  Usefulness  of  the  Friendly  Protector  Function  Performance  Measures 
Consider  the  Friendly  Protector  performance  measures  listed  below.  Rate  each  one  in  terms  of  how  useful  it 
can  be  to  you  in  your  job. 


"MEASURE 

I  Friendly  Protector  FPMs  full  imckM 

1  tracks  IDed 

not  useful 

1 

2 

somewhat 

useful 

3 

4 

very  useful 

5 

%  tracks  IDed  correct 

1 

*> 

3 

4 

5 

tracks  IDed  early 

1 

2 

3 

4 

5 

r«  tracks  IDed  within  window 

1 

3 

4 

5 

I 

i  tracks  IDed  late 

1 

3 

4 

S 

average  delay  to  ID 

1 

3 

4 

5 

neglected 

1 

-t 

3 

4 

5 

i  Friendly  Protector  FPMs  (friends  onlyl 

i 

1  ^  friends  IDed 

i 

^  1 

2 

3 

4 

5 

i 

%  friends  IDed  correct 

1 

-> 

3 

4 

5 

*1  friends  IDed  early 

1 

‘ ) 

3 

4 

5 

|  %  friends  IDed  within  window 

I 

2 

3 

4 

5 

|  "*  friends  IDed  late 

1 

3 

4 

5 

average  delay  to  ID  friends 

1 

2 

3 

4 

5 

"5 
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MEASURE 

FritndlY  Projector  FPMs  (friends  only) 

%  friends  IDed 

not  useful 

1 

2 

somewhat  j 

useful 

3  4 

very  useful 

5 

Friendly  Protector  FPMs  (hot Hies  only) 

- 

%  hosllles  IDed 

i 

3  4 

5 

%  hosllles  IDed  correct 

i 

3  4 

5 

%>  hosllles  IDed  early 

i 

2 

3  4 

S 

%  hosllles  IDed  within  window 

i 

3  4 

5 

' 

%  hosllles  IDed  lale 

i 

3  4 

S 

overage  delay  to  ID  hosllles 

i 

2 

3  4 

5 

"t  hosllles  threatening 

i 

*> 

3  4 

S 

Rate  the  Frequency,  Timeliness,  and  Format  of  the  Friendly  Protector  FPMs 

Now,  consider  the  Friendly  Protector  function  performance  measures  as  a  group.  Rate  them  in  terms  of  the 
frequency,  timeliness,  and  format  in  which  you  would  need  them. 

Frequency:  1  2 

quarterly  or  monthly 
less 

3 

weekly 

4 

once  a  day 

5 

several  limes  a  day 

Timeliness:  l  ,2 

week  or  l  week 

longer 

3  * 
a  day 

4 

1  hour 

5 

10  min.  or  less 

Format:  t  2 

printout  disk/tape 

3 

interactive 
display 
(data  tables) 

4 

Inleracllve 

display 

(graphics) 

5 

expert  system 
critique  w !  replay 
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RATING  FORM  FOR  PCOFT/TPT  WEAPONS  CONTROLLER 
CANDIDATE  FUNCTION  PERFORMANCE  MEASURES 

Rating  the  Weapons  Controller  Function  Performance  Measures 
In  this  section,  consider  the  PTOS  PAC  Weapons  Controller  function  performance  measures.  As  before,  rate 
information  provided  by  each  measure  on  the  dimension  of  usefulness  first.  Then,  rate  the  Weapons  Controller 
function  performance  measures  as  a  group  in  terms  of  frequency,  timeliness,  and  format.  Remember  to  rate 
each  item  as  it  relates  to  your  information/data  needs.  Feel  free  to  refer  to  the  descriptions  of  the  rating 
dimensions  and  to  the  briefing  slides  to  refresh  your  memory  of  the  function  performance  measures. 


Rate  the  Usefulness  of  the  Weapons  Controller  Function  Performance  Measures 
Consider  the  Weapons  Controller  function  performance  measures  listed  below.  Rate  each  one  in  terms  of 
how  useful  it  can  be  to  you  in  your  job. 


MEASURE 


Weapons  Controller  FPMs 

not  useful 

somewhat 

useful 

very  useful 

j 

%  hostile*  engaged 

1 

"» 

3 

4 

5 

i 

%  hosliles  killed 

1 

2 

3 

4 

S 

kill  ratio 

1 

2 

3 

4 

S 

early  engagements 

1 

2 

3 

4 

5 

1 

%  engagements  within  window 

... 

1 

2 

3 

4 

5 

1 

%  engagements  late 

1 

2 

3 

4 

5 

average  delay  to  engage 

l 

2 

3 

4 

5 

avg  engage  delay  from  window  end  (late  engages) 

l 

3 

4 

3 

\  average  ATC  at  launch 

\ 

1 

-> 

3 

4 

5 

!  %  failures  due  to  weapons  controller 

1 

2 

3 

4 

5 

I  avetage  launches  per  engaged  track 

1 

2 

3 

4 

> 

Please  continue  to  next  page. 
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Rate  the  Frequency,  Timeliness,  and  Format  of  the  Weapons  Controller  FP^ls 

Now,  consider  the  Weapons  Controller  function  performance  measures  us  a  group.  Rale  them  in  terms  of  the 
frequency,  timeliness,  and  format  in  which  you  would  need  them. 


Frequency: 

l 

quarterly  or 
less 

2 

monthly 

3 

weekJy 

4 

once  a  day 

5 

several  times  a  day 

Timeliness: 

1 

week  or 
longer 

2 

1  week 

3 

a  day 

4 

l  hour 

5 

10  min-  or  less 

Format: 

1 

printout 

■J 

disk  Lip* 

3 

interactive 

display 

(da La  tables) 

4 

interactive 

display 

(graphics) 

s 

expert  system 
critique  w,  replay 
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RATING  FORM  FOR  PCOFT/TPT  CANDIDATE  TASK  PERFORMANCE 

MEASURES 


Rating  the  Task  Function  Performance  Measures 
In  this  section,  consider  the  PTOS  PAC  task  performance  measures.  As  before,  rate  information  provided  by 
each  measure  on  the  dimension  of  usefulness  first.  Then,  rate  the  task  performance  measures  as  a  group  in 
terms  of  frequency,  timeliness,  and  format.  Remember  to  rate  each  item  as  it  relates  to  your  information/data 
needs.  Feel  free  to  refer  to  the  descriptions  of  the  rating  dimensions  and  to  the  briefing  slides  to  refresh  your 
memory  of  the  task  performance  measures. 


Rate  the  Usefulness  of  the  Task  Performance  Measures 

Consider  the  PTOS  PAC  task  performance  measures  listed  below.  Rate  each  one 
can  be  to  you  in  your  job. 

in  i 

terms  of  how  useful  i 

MEASURE 

Swiich  Action-Based  TPMs 

all  (racks  hooked 

not  useful 

r  2 

somewhat 

useful 

3 

4 

very  useful 

5 

*«  scripted  friends  hooked 

i 

2 

3 

4 

5 

r«  scripted  hosliles  hooked 

i 

2 

3 

4 

5 

**  all  tracks  IFFed 

l 

2 

3 

4 

5 

*e  scripted  friends  IFFed 

i 

2 

3 

4 

5 

®i  scripted  hosliles  IFFed 

i 

2 

3 

4 

5 

average  number  swilch  actions  per  track 

! 

2 

3 

4 

5 

Alert-Based  TPMs 

alerts  displayed 

1 

2 

3 

4 

5 

ct  alerls  expired 

1 

3 

4 

5 

Or  alerls  lost 

1 

3 

4 

5 

average  delay  to  display  an  alert 

1 

2 

3 

4 

5 

average  delay  to  acknowledge  an  alert 

1 

*> 

3 

4 

5 

Please  continue  to  next  page. 
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Rate  the  Frequency,  Timeliness,  and  Format  of  the  Task  Performance  Measures 

Now,  consider  the  task  performance  measures  as  a  group.  Rate  them  in  terms  of  the  frequency,  timeliness, 
and  format  in  which  you  would  need  them. 


Frequency: 

t 

quarterly  or 
less 

V 

4> 

monthly 

3 

weekly 

4 

once  a  day 

5 

several  limes  a  day 

Timeliness: 

l 

week  or 
longer 

2 

1  week 

3 

a  day 

4 

1  hour 

s 

10  min.  or  less 

Format: 

1 

printout 

■> 

desk,  tape 

3 

Interactive 
display 
(data  tables) 

4 

interactive 

display 

(graphics) 

S 

expert  system 
critique  v*,  replay 

10 
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RATING  FORM  FOR  PCOFT/TPT  CANDIDATE  DETAILED 
PERFORMANCE  HISTORIES 

Rating  the  Detailed  Performance  Histories 

In  this  section,  consider  the  PTOS  PAC  detailed  performance  histories.  As  before,  rate  information  provided 
by  different  types  of  information  on  the  dimension  of  usefulness  first.  Then,  rate  the  detailed  performance 
histones  in  terms  of  frequency,  timeliness,  and  format.  Remember  to  rate  each  item  as  it  relates  to  your  infor¬ 
mation/data  needs.  Feel  free  to  refer  to  the  descriptions  of  the  rating  dimensions  and  to  the  briefing  slides  to 
refresh  your  memory  of  the  detailed  performance  histories. 


Rate  the  Usefulness  of  the  Detailed  Performance  Histories 

Consider  the  PTOS  PAC  detailed  performance  history  information  categories  listed  below.  Rate  each 
category  in  terms  of  how  useful  it  can  be  to  you  in  your  job. 


INFORMATION  CATEGORY 

not  useful 

somewhat 

useful 

very  u  so  In  1 

task  window 

i  : 

3 

4 

5 

operator  hooks 

1  2 

3 

4 

5 

operator  switch  actions 

t  : 

3 

4 

5 

track  ID  history 

t  : 

3 

4 

5 

system  events 

t  : 

3 

4 

5 

track  events 

t  : 

3 

4 

5 

Rate  the  Frequency,  Timeliness,  and  Format  of  the  Detailed  Performance  Histories 

Now,  rate  the  detailed  performance  histories  in  terms  of  the  frequency,  timeliness,  and  format  in  which  you 
would  need  them. 


Frequency: 

i 

quarterly  or 
less 

■» 

monthly 

3 

weekly 

4 

once  a  day 

several  times  a  day 

Timeliness: 

l 

week  or 
longer 

i  week 

3 

a  day 

4 

l  hour 

5 

10  min.  or  less 

Format: 

1 

printout 

disk/tnpe 

3 

interactive 
display 
(data  tables) 

4 

interactive 
display 
(graphics  i 

5 

expert  system 
critique  replay 
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COMMENTS  SECTION 

In  this  section  we  ask  you  to  make  any  comments  you  would  like  about  any  aspect  of  the  PAC.  Possible 
topics  include  uses  for  the  PAC  not  already  specified,  changes  to  measures  currently  used,  suggestions  for  new 
scores  and  measures,  and  different  formats  for  displaying  and  presenting  PAC  data. 


12 
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APPENDIX  C 


Candidate  Task  Lists 


14E 

Supervise  Emplacement  of  Information  Coordination  Central 
Task  Number  Task  Description 

01-0401. 05-TBD  Energize  the  Information  Coordination  Central 


Initialization 


01-0401.05-0045 

01-0401.05-0046 


01  0401.05-0047 


01-0401.05-0048 

01-0401.05-0049 

01-0401.05-0050 


Task.  Description 

Monitor  data  modem  bias  adjustment  in  the  Information  Coordination 
Central 

Supervise  manual  tactical  software  initialization  in  the  Information  Coor¬ 
dination  Central  (ICC) 

Supervise  automatic  tactical  software  initialization  in  the  Information  Coor 
dination  Central  (ICC) 

Supervise  initialization  of  software  using  last  prior  data  base  (LPDB) 
Supervise  recovery  operations  in  the  Information  Coordination  Central 
(ICC) 

Supervise  alignment  of  the  Patriot  firing  battery 


Tactical  Operations 


Task  Number 
01-0401.05-0087 

01-0401.05-0088 

01-0401.05-0089 

01-0401.05-0090 

01-0401.05-0091 

01-0401.05-0092 


Task  Description 

Perform  power-up/power-down  procedures  on  the  Information  Coordina¬ 
tion  Central  (ICC) 

Perform  rapid  or  emergency  power-down  procedures  in  the  Information 
Coordination  Central 

Perform  protection  of  friendly  aircraft  entering  the  Battalion  Area  of 
Responsibility  in  the  Information  Coordination  Central  (ICC) 

Perform  engagement  of  targets  from  the  Information  Coordination  Central 
(ICC) 

Monitor  tactical  situations  and  status  of  battalion  elements  and  plan  bat¬ 
talion  deployment  in  response  to  tactical  requirements 
Perform  alternate  deployment  activation  in  the  Information  Coordination 
Central  (ICC) 


C-i 


Task  Number 
01-0401.05-0093 

01-0401.05.0094 

01-0401.05-0096 

0 1-040 1.05-TBD 

01-0401.05-TBD 

01-0401.05-0132 

01-0401.05-0417 

01-0401.05-0418 

01-0401.05-0149 

01-0401.05-0150 


Task  Description 

Perform  a  fire  platoon  initialization  support  request  in  the  Information 
Coordination  Central  (ICC) 

Perform  fire  platoon  data  base  comparison  in  the  Information  Coordina¬ 
tion  Central  (ICC) 

Monitor  data  modem  operation  in  the  Information  Coordination  Central 
(ICC) 

Send  free  form  message  from  the  Information  Coordination  Central 
(ICC)(01-0401.05-0100) 

Perform  integration  procedures  with  the  Brigade  AN/TSQ-73  Missile 
Minder  in  the  Information  Coordination  Central  (ICC) 

Supervise  electro  magnetic  pulse  recovery  in  the  Information  Coordination 
Central  (ICC) 

Supervise  the  Firing  Battery  Air  Defense  Battle  in  Centralized  Mode 
Supervise  the  Firing  Battery  Air  Defense  Battle  In  Decentralized  Mode 
Supervise  the  Firing  Battery  Air  Defense  in  Centralized  Mode 
Supervise  the  Firing  Battery  Air  Defense  in  Autonomous  Mode 


24T 


Performing  System  Initialization 


Task  Number 
441-083-1407 

441-083-1408 

441-083-1124 

441-083-1409 

441-083-1410 

441-084-1125 


Task  Description 

Perform  as  crew  member  No.l  (MSl)during  Engagement  Control  Station 
(ECS)  initialization 

Perform  as  crew  member  No.  2  (MS3)during  Engagement  Control  Station 
(ECS)  initialization 

Perform  as  crew  member  No.  3(MS2)during  Engagement  Control  Station 
(ECS)initialization 

Perform  as  crew  member  No.  l(MSl)during  Information  Coordination 
Central  initialization 

Perform  as  crew  member  No.  2  (MS3)during  Information  Coordination 
Central  (ICC)  initialization 

Perform  as  crew  inember  No.  3(MS2)during  Information  Coordination 
Central  (ICC)initialization 


Perform  Patriot  System  Operational  Procedures 


Task  Number 
441-083-1471 
441-083-1472 


Task  Description 
Activate  Fire  unit 

Activate  Information  Coordination  Central  (ICC) 


r  o 


441-083-1473  Change  configuration  from  on-line  to  primary/secondary  network  Information 
Coordination  Central  (ICC) 

441-083-1474  Deactivate  fire  unit 

441-083-1475  Deactivate  Information  Coordination  Central  (ICC) 

441-083-1476  Engage  jammers-Engagement  Control  Station  (ECS) 

441-083-1477  Engage  tactical  ballistic  missile-Engagement  Control  Station  (ECS) 

441-083- 1478  Engage  targets-Engagement  Control  Station  (ECS) 

441-083-1479  Evaluate  pre-engagement  data-Engagement  Control  Station  (ECS) 
441-083-1480  Evaluate  pre-engagement  data-Information  Coordination  Central  (ICC) 
441-083-1481  Initiate  jammer  engagements-Information  Coordination  Central  (ICC) 
441-083-1482  Initiate  target  engagements-Information  Coordination  Central  (ICC) 
441-084-1114  Perform  compulsory  safety  procedures 

441-084-1130  Perform  data  modem  operations 

441-083-1483  Perform  fire  platoon  data  base  comparison-information  Coordination  Central 
(ICC) 

441-083-1484  Perform  fire  platoon  initialization  support-information  Coordination  Central 
(ICC) 

441-083-1485  Perform  fire  unit  to  fire  unit  operations-Engagement  Control  Station 
441-083-1486  Perform  friendly  protect  -Engagement  Control  Station  (ECS) 

441-083-1487  Perform  friendly  protect  -  Information  Coordination  Central  (ICC) 

441-083-1488  Perform  missile  hazard/  misfire  procedures  Engagement  Control  Station 
(ECS) 

441-083-1489  Perform  mode  transition  procedures-Engagement  Control  Station  (ECS) 
441-083-1490  Perform  reinitialization  in  the  Engagement  Control  Station  (ECS) 
441-083-1491  Perform  saturation  alleviation  procedures-Engagement  Control  Station  (ECS) 
441-083-1492  Perform  system  reorientation  and  clutter  map  update  (CMUP)  Engagement 
Control  Station  (ECS) 

441-083-1493  Verify  Identification  Friend  or  Foe  (IFF)  operabilitv-Engagement  Control 
Station  (ECS) 


APPENDIX  D 


User  Group  Ratings  of  Usefulness  of  Individual  ARI  PAC  Measures 


Table  D-l 

Average  Usefulness  Ratings  of  Individual  Mission  Performance  Measures  by  User  Groups 


MPMs 


Defense 
of  Assets 

Friendly 

Protection 

DOTD 

5.0 

5.0 

5.0 

5.0 

DESCSD 

4.8 

4.8 

5.0 

4.5 

PATTRN-14E 

4.3 

4.4 

4.5 

3.9 

PATTRN-24T 

2.9 

2.7 

2.4 

2.7 

PATTRN-D 

4.8 

5.0 

4.4 

3.8 

CATD 

4.7 

4.3 

5.0 

4.7 

32  OTT 

4.5 

5.0 

5.0 

4.3 

32  T/E 

4.8 

4.5 

5.0 

3.8 

11TH  BDE 

4.7 

4.5 

4.9 

4.1 

6TH  BDE 

4.7 

4.3 

4.7 

3.3 

4/43  ADA 

4.0 

5.0 

5.0 

5.0 

4/7  ADA 

4.8 

4.5 

4.6 

4.2 

HAWK 

5.0 

5.0 

4.3 

4.3 

Means 

4.5 

4.4 

4.5 

4.0 

D-1 


Table  D-2 


Average  Usefulness  Ratings  of  Friendly  Protector  Function  Performance  Measures  (All 
Aircraft) 


Group 

% 

IDed 

% 

IDed 

Corr 

% 

IDed 

Earlv 

FPMs 

% 

IDed 
in  Win 

% 

IDed 

Late 

Avg 
Delay 
to  ID 

% 

Neglected 

DOTD 

5.0 

5.0 

4.0 

4.5 

4.5 

3.5 

4.0~ 

DESCSD 

5.0 

5.0 

4.8 

4.8 

4.8 

4.8 

4.8 

PATTRN-14E 

4.3 

4.5 

4.0 

3.8 

4.3 

3.1 

4.3 

PAT  TRN-24T 

3.7 

4.1 

2.6 

2.7 

3.0 

2.4 

2.9 

PAT  TRN-D 

4.8 

5.0 

3.0 

4.4 

5.0 

3.0 

4.6 

CATD 

4.3 

4.7 

3.3 

3.7 

4.3 

3.3 

5.0 

32  OTT 

4.5 

4.8 

3.5 

4.8 

4.8 

3.3 

4.0 

32  T/E 

4.8 

5.0 

3.3 

4.5 

4.5 

4.0 

4.8 

1 1TH  BDE 

3.9 

4.9 

3.5 

4.4 

4.2 

4.0 

4.1 

6TH  BDE 

4.3 

4.3 

3.0 

3.7 

4.3 

3.7 

4.3 

4/43  ADA 

5.0 

5.0 

2.0 

3.5 

3.5 

2.0 

4.0 

4/7  ADA 

3.8 

4.7 

3.1 

4.1 

4.0 

3.6 

4.1 

HAWK 

3.7 

4.7 

2.0 

4.3 

3.7 

3.7 

4.3 

Means 

4.2 

4.7 

3.3 

4.1 

4.2 

3.5 

4.2 

D-2 


Table  D-3 


Average  Usefulness  Ratings  of  Friendly  Protector  Function  Performance  Measures 
(Friends  Only) 


FPMs 


Group _ 

% 

IDed 

% 

IDed 

Corr 

% 

IDed 

Early 

% 

IDed 
in  Win 

% 

IDed 

Late 

Avg 
Delay 
to  ID 

% 

at  Risk 

DOTD 

5.0 

4.5 

4.0 

4.0 

4.5 

4.5 

4.0 

DESCSD 

5.0 

4.3 

4.8 

4.8 

4.8 

4.8 

5.0 

PATTRN-14E 

4.0 

4.5 

4.1 

3.6 

3.8 

3.8 

3.9 

PATTRN-24T 

3.4 

3.9 

2.7 

2.7 

3.0 

3.0 

2.9 

PATTRN-D 

5.0 

5.0 

3.0 

4.0 

5.0 

5.0 

4.4 

CATD 

4.3 

5.0 

3.3 

3.3 

4.3 

4.3 

4.3 

32  OTT 

4.5 

4.8 

3.5 

4.3 

4.8 

4.8 

4.8 

32  T/E 

4.8 

5.0 

3.3 

4.5 

4.0 

4.0 

4.8 

11TH  BDE 

4.5 

4.9 

3.6 

4.5 

4.2 

4.2 

4.4 

6TH  BDE 

4.0 

4.0 

3.3 

3.7 

4.3 

4.3 

3.3 

4/43  ADA 

5.0 

5.0 

2.0 

3.5 

3.5 

3.5 

4.0 

4/7  ADA 

3.6 

4.6 

3.5 

4.1 

4.0 

4.0 

3.7 

HAWK 

4.0 

4.7 

2.0 

3.3 

3.3 

3.3 

4.0 

Means 

4.2 

4.6 

3.4 

4.0 

4.1 

3.6 

4.0 

Table  D-4 


Average  Usefulness  Ratings  of  Friendly  Protector  Function  Performance  Measures 
(Hostiles  Only) 


Group _ 

% 

IDed 

% 

IDed 

Corr 

% 

IDed 

Earlv 

FPMs 

% 

IDed 
in  Win 

% 

IDed 

Late 

Avg 
Delay 
to  ID 

% 

Threatening 

DOTD 

4.5 

4.5 

4.0 

4.0 

4.0 

4.5 

4.5 

DESCSD 

4.8 

4.8 

4.8 

4.8 

4.8 

4.8 

5.0 

PATTRN-14E 

4.5 

4.5 

4.0 

3.9 

4.5 

3.5 

4.3 

PAT  TRN-24T 

4.0 

4.0 

2.6 

2.6 

3.1 

2.6 

3.1 

PATTRN-D 

5.0 

5.0 

3.0 

4.0 

4.8 

3.4 

4.6 

32  0TT 

4.5 

4.8 

4.0 

4.5 

4.8 

4.0 

4.0 

32  T/E 

4.8 

5.0 

3.3 

4.5 

4.8 

3.8 

4.8 

CATD 

4.3 

4.7 

4.0 

3.7 

4.3 

4.0 

4.7 

1 1TH  BDE 

4.5 

4.9 

3.9 

4.5 

4.3 

4.1 

4.7 

6TH  BDE 

4.3 

5.0 

4.0 

3.7 

4.3 

3.7 

4.0 

4/43  ADA 

5.0 

5.0 

3.0 

3.0 

3.5 

4.0 

5.0 

4,7  ADA 

4.1 

4.8 

3.8 

4.3 

3.9 

3.6 

4.3 

HAWK 

3.7 

4.7 

2.7 

3.3 

3.7 

3.7 

4.3 

Means 

4.4 

4.7 

3.7 

4.1 

4.2 

3.7 

4.4 

D-4 


Table  D-5 


Average  Usefulness  Ratings  of  Weapons  Controller  Function  Performance  Measures 


FPMs 


Group 

% 

Hos 

Eng 

% 

Hos 

Kil 

Kill 

Ratio 

% 

Early 

Eng 

%  % 

in  Win  Eng 
Eng _ Late 

Avg  Avg 
Delay  Lnch 
Eng  ATC 

% 

W.C. 

Fail 

Avg 

Lnch 

/Trk 

DOTD 

5.0 

5.0 

5.0 

4.0 

4.5 

4.0 

4.0 

4.0 

5.0 

4.0 

DESCSD 

5.0 

5.0 

5.0 

4.8 

4.8 

4.8 

4.8 

4.8 

5.0 

4.8 

PATTRN-14E 

4.1 

4.3 

3.8 

4.1 

3.8 

3.4 

3.8 

3.9 

4.1 

3.8 

PATTRN-24T 

4.0 

3.1 

2.9 

2.7 

3.0 

2.3 

2.6 

3.3 

3.1 

3.3 

PATTRN-D 

5.0 

4.6 

3.4 

3.0 

4.6 

3.8 

4.6 

1.6 

4.8 

3.2 

CATD 

4.0 

4.0 

3.7 

3.3 

4.0 

3.7 

4.3 

3.0 

4.3 

3.7 

32  OTT 

5.0 

5.0 

4.5 

3.5 

4.5 

3.8 

3.8 

4.0 

4.8 

4.5 

32T/E 

4.5 

4.5 

4.5 

3.2 

4.5 

3.8 

4.0 

4.0 

4.8 

4.3 

1 1TH  BDE 

4.5 

4.2 

4.8 

3.6 

3.9 

3.9 

3.5 

3.5 

4.6 

4.1 

6TH  BDE 

4.7 

4.3 

3.0 

3.3 

4.3 

3.7 

4.7 

4.0 

3.3 

3.7 

4/43  ADA 

5.0 

5.0 

4.5 

2.0 

5.0 

2.5 

2.5 

1.5 

4.5 

3.5 

4/7  ADA 

4.2 

4  .4 

4.0 

3.5 

4.1 

3.4 

3.7 

3.2 

4.4 

3.2 

HAWK 

4.0 

3.7 

3.3 

2.0 

4.0 

4.3 

4.0 

4.7 

4.3 

3.0 

Means 

4.4 

4.3 

4.0 

3.8 

4.1 

3.6 

3.8 

3.4 

4.4 

3.7 

D-5 


Table  D-6 


Average  Usefulness  Ratings  of  Switch  Action-Based  Task  Performance 
Measures 


Switch  Action  TPMs 

%  %  %  %  %  %  Avg. 

Trks  Frnds  Host  Trks  Frnds  Host  Sws 


[CJTi 

IFFed 

■I323SK 

■m 

sStn!«!R3«l 

ISiSiSiImiI 

-Acts 

DOTD 

4.5 

4.5 

4.5 

4.5 

4.5 

4.5 

4.5 

DESCSD 

5.0 

5.0 

5.0 

4.5 

4.5 

4.5 

4.5 

PATTRN-14E 

4.1 

4.0 

4.4 

4.1 

4.1 

4.5 

2.6 

PAT  TRN-24T 

2.7 

2.3 

3.0 

2.4 

2.1 

2.6 

2.6 

PAT  TRN-D 

5.0 

4.8 

5.0 

4.0 

4.2 

4.4 

2.2 

CATD 

4.0 

3.7 

3.7 

4.0 

3.0 

3.7 

3.0 

32  OTT 

4.5 

4.3 

4.3 

4.5 

4.3 

4.5 

4.0 

32T/E 

4.5 

4.5 

4.5 

4.0 

4.0 

4.5 

3.5 

1 1TH  BDE 

4.5 

3.8 

3.8 

4.1 

3.8 

3.9 

3.5 

6TH  BDE 

3.0 

3.0 

3.0 

3.0 

3.0 

2.7 

2.7 

4/43  ADA 

5.0 

4.5 

4.5 

4.0 

5.0 

5.0 

3.5 

4/7  ADA 

3.5 

3.7 

3.9 

2.9 

3.3 

3.5 

3.0 

HAWK 

4.0 

3.7 

3.7 

4.0 

4.0 

4.0 

3.7 

Means 

4.0 

3.8 

4.0 

3.7 

3.7 

3.9 

3.2 

D-6 


Table  D-7 


Average  Usefulness  Ratings  of  Alert-Based  Task  Performance  Measures 


Alert  TPMs 


Gro.;p 

% 

Alerts 

I  nst 

% 

Alerts 

Expired 

% 

Alerts 

Displayed 

Avg. 
Delay  to 
Display 

Avg. 

Delay  to 
Acknowledge 

DOTD 

4.5 

4.5 

4.5 

4.5 

4.5 

DESCSD 

4.5 

4.5 

4.5 

4.5 

4.5 

PATTRN-14F 

3.9 

3.8 

3.5 

3.6 

4.1 

PATTRN-24T 

2.9 

2.4 

2.6 

2.7 

3.3 

PATTRN-D 

3.0 

3.2 

3.0 

3.2 

3.6 

CATD 

4.3 

4.3 

4.3 

4.0 

4.7 

32  OTT 

4.5 

4.0 

4.5 

3.8 

4.0 

32T/E 

4.0 

4.2 

4.2 

4.0 

4.8 

11TH  BDE 

4.2 

4.0 

4.2 

3.6 

4.0 

6TH  BDE 

3.7 

3.7 

4.3 

3.7 

4.0 

4/43  ADA 

4.5 

4.5 

4.5 

3.5 

4.0 

4/7  ADA 

3.8 

3.2 

3.3 

3.5 

3.7 

HAWK 

1.0 

1.0 

1.0 

1.0 

1.0 

Means 

3.8 

3.5 

3.6 

3.5 

3.9 

D-7 


Table  D-8 


Average  Usefulness  Ratings  of  Detailed  Performance  History  Data 


DPHs 


Group 

Window 

Hooks 

Switch 

Actions 

ID 

Historv 

System 

Actions 

Track 

Actions 

DOTD 

4.0 

4.0 

4.0 

4.0 

4.0 

4.0 

DESCSD 

4.0 

4.0 

4.0 

5.0 

4.0 

5.0 

PATTRN-14E 

3.4 

4.1 

4.0 

4.7 

4.0 

4.1 

PAT  TRN-24T 

1.7 

3.4 

3.7 

4.4 

3.7 

3.6 

PAT  TRN-D 

3.4 

3.0 

2.6 

2.8 

2.6 

3.0 

CATD 

4.0 

3.7 

4.0 

4.7 

4.0 

5.0 

32  0TT 

4.5 

4.5 

4.5 

4.3 

4.5 

4.2 

32  T/E 

4.5 

4.0 

3.8 

4.0 

3.8 

4.8 

1 1TH  BDE 

4.1 

3.9 

3.9 

4.2 

3.9 

4.2 

6TH  BDE 

4.3 

2.7 

4.0 

3.7 

4.0 

4.0 

4/43  ADA 

4.5 

4.0 

4.5 

4.5 

4.5 

5.0 

4/7  ADA 

3.5 

3.4 

3.6 

3.7 

3.6 

3.8 

HAWK 

3.3 

3.0 

3.3 

3.3 

3.3 

3.7 

Means 

3.7 

3.7 

3.8 

4.1 

3.8 

4.1 

APPENDIX  E 


Organization 

DOTD: 

DESCSD: 

32T/E: 

PATTRN-24T: 

PATTRN-24T: 

PATTRN-24T: 

PATTRN-24T: 


PAT  TRN-24T: 

PATTRN-24T. 


User  Group  Comments  On  ARI  PAC 


Comments 

Must  evaluate  ECM  EW9A20/Track  technique.  For  HAWK  what  action  opera¬ 
tion  makes  depends  on  Jammer  and  Equipment  indications.  Must 
evaluate  target  reactive  or  evasive  maneuvers  as  process  in  determining 
friend/hostile.  Evaluate  correct  action  taken  in  alert.  Evaluation  must 
consider  trackload/saturation. 

Add  to  MPM,  Kills  Before  Ordnance  Release  (KBOR),  it  is  important  to  kill  the 
hostile  before  he  can  damage  our  assets  so  KBOR  is  an  important  con¬ 
sideration  in  assessing  performance. 

The  system  needs  to  be  fast  and  accurate.  Scenarios  may  be  run  6-8  times  a  day. 

The  concept  presented  sure  to  have  considerable  potential.  However,  a 
demonstration  conducted  on  the  OTT  should  certainly  be  conducted 
before  any  final  decision  is  made. 

Simple  and  Task  oriented  scenarios.  The  ability  for  replay  and  (Printout) 
needed  for  Instructor  Feedback. 

A  system  to  score  the  S/I  actions  used  by  the  operator  in  either  the  TCO  or 
TCA  position  would  be  useful  as  well  as  recalling  the  proper  tabs  for 
selected  functions  called  for  by  the  instructor. 

In  a  24T10  Basic  Course,  we  are  only  concerned  if  the  student  understands 
basic  performance  of  certain  critical  tasks.  We  are  not  concerned  if  the 
student  performs  in  a  certain  time  period  but  if  he  knows  how  to  use  tabs, 
engage  measures,  IFF  measures  etc....  to  include  Initialization,  Radar 
Mapping  &  Command  Plan  tabs.  To  evaluate  a  basic  soldier  without  fur¬ 
ther  development  of  the  software  would  not  be  advisable. 

Student  switch  actions,  initialization  ECS  &  ICC,  Radar  mapping  com¬ 
mand  plan,  TCO  switch  actions  for  critical  task,  TCA,  TD,  TDA,  Instruc¬ 
tor  console  is  utilized  by  74D,  computer  operator,  not  a  24T. 

I  think  the  concept  is  great  and  could  be  a  useful  tool,  but  1  don't  think  the 
study  was  conducted  properly,  i.e.,  no  input  from  24T  Instructor  to  deter¬ 
mined  how  this  would  tie  into  the  OTT  Software  and  how  much  time  it 


E-i 


Organization 

PATTRN-D: 

PATTRN-D: 

PATTRN-D: 

PATTRN-D: 

PAT  TRN-D: 

1 1TH  BDE: 

1 1TH  BDE. 

1 1TH  BDE 

1 1TH  BDE: 


would  take  to  grade  a  student  and  review  the  student’s  grades  with  them.  "Note"  the 
structor  will  not  have  access  to  instructor  console. 

Number  of  alerts  generated  must  be  reduced  before  any  operator  can  pos¬ 
sibly  keep  up  with  them,  especially  at  the  ICC.  As  it  is  now,  maybe  we 
should  score  students  only  on  H.E.  alerts. 

Under  "Rating  Dimensions"  Expert  System  Replay  should  be  selective  by 
showing  those  hard  or  specific  errors  committed  by  students. 

FORMAT  5.  I  would  rather  have  immediate  feedback  to  student  versus  a 
replay  such  as  a  highlighted  block  on  the  target  the  error  was  on  and  the 
tabular  display  affected.  Example  track  001  highlighted  and  the  Eng 
Data  Tab  with  a  negative  TLL  highlighted. 

%  Friends  Engaged  -  More  emphasis  on  TOI.D-IN  TRACK  ID.  Certainly 
some  of  tile  current  ID  Alerts  (ICC). 

ID  Window  from  cross  FSCL  to  +  30  sec.  Additional  ID  windows  when  ID 
history  change  will  effect  ID  of  track  IAW  EDWA.  Engage  window 
from  Threat  Level  below  9  to  TLL  =  0. 

PAC  sounds  great!  I  look  forward  to  working  with  it  in  the  future.  Especially 
if  an  Expert  System  Critique  with  display  could  be  given  so  that  the 
operator  could  play  back  certain  segments  or  windows  of  the  air  battle. 

But  only  if  these  segments  can  be  accurate  as  to  the  specific  point  in  air- 
battle  that  the  operator  wishes  to  see  and  evaluate.  GOOD  BRIEFING 

Sounds  like  a  good  program,  need  to  see  it  in  action  in  order  to  give  a  good 
evaluation  of  its  effectiveness. 

There  is  a  lot  of  information  being  considered  some  very  useful  some  not  as 
useful,  but  it  is  obvious  that  the  new  evaluation  system  will  be  much  more 
helpful  in  training.  After  school  the  PCOFT  is  rarely  used,  TPT’s  are 
run  in  the  ECS  along  with  OTM’s.  Attrition  and  Defense  of  Assets  are 
often  two  different  missions,  and  the  scores  could  differ  to  reflect  these 
missions.  In  some  cases  100%  Attrition  would  increase  the  Waste  of 
Resources  due  to  low  P.K.  or  tail  chasing. 

The  concept  is  exceptionally  valid.  The  concern,  however,  is  priorities  set  for 
its  use.  The  System  needs  to  be  designed  for  use  at  the  Battalion.  It 


E-2 


Organization 


Comments 


4/43  ADA: 

4/7  ADA: 

4/7  ADA: 

4/7  ADA: 

4/7  ADA: 

4/7  ADA: 

HAWK: 


must  be  capable  of  adjustment  to  the  Battalion’s  specific  mission  needs.  Most  need 
system  capable  of  evaluating  a  joint  exercise/live  aircraft  or  adapt  the  con¬ 
cept  to  live  air  trainer.  This  is  very  futuristic  but  feasible.  Good  Presen¬ 
tation.  Concept  will  work  and  help  unit  readiness  without  a  doubt. 

PAC  needs  to  be  incorporated  into  32d  350-29  (Basic,  Senior  Master)  levels  of 
training  for  Air  Battle  Management  (ABM).  Evaluation  of  performance 
needs  to  be  a  team  TCO/TCA  score  analysis  and  an  Individual  Score  of 
Performance.  Basic,  Senior  Master  levels  of  training  should  be  used  in 
establishing  standards  and  conditions  for  tasks  in  ABM  training  software. 

This  would  assist  in  the  analysis  of  what  factors  to  evaluate/score  for  fol¬ 
low  on  training. 

Will  there  be,  at  anytime  in  the  future,  a  TFT  (Netted)  for  use  with  Hawk/Patriot? 
Not  a  stand-alone  Hawk/Patriot. 

I  don’t  think  the  government  should  invest  in  this  type  of  trainer.  We  are  better 
off  training  TCO’s  and  TCA’s  to  standards  with  evaluators  and  LAT.  A 
software’s  system  cost  compared  to  the  benefit  it  will  have  on  TCO,  TCA 
proficiency  will  not  be  worth  the  expense  to  the  government. 

I  completed  this  form  based  on  my  knowledge  from  the  PCOFT  at  O.B.L.  I  am 
not  at  this  time  a  qualified  TCO. 

Everything  looks  good  on  paper,  we  need  to  apply  this  knowledge  to  the  System 
and  use  it  in  netted  scenarios.  To  determine  its  validity  for  air  battle 
management. 

I  would  like  to  see  this  available  for  the  ICC/ECS.  The  PCOFT  isn't  as  function¬ 
al  as  the  real  thing. 

Personnel  History  -  i.e.:  E2  scores,  Education  level,  climatic  conditions,  time  on 
system.  How  many  hours  worked  with  at  rest.  What  duties  performed 
before  entering  into  task. 


E-3 


APPENDIX  F 


Cross  Reference  of  PAC  Summary  File  Data  Elements 

To 

PAC  Performance  Measures 

Three  data  tables  are  used  to  collect  and  compute  ARI 
PAC  summary  performance  measures:  the  threat  group  summary 
table,  the  asset  penetration  table,  and  the  alert  message 
summary  table.  The  threat  group  summary  table  is  used  to 
compute  all  measures  except  defense  of  assets  and  the  alert- 
based  TPMs.  The  asset  penetration  table  is  used  to  compute 
defense  of  assets.  The  alert  message  summary  table  is  used 
to  compute  alert-based  TPMs.  Within  the  tables,  there  are 
two  kinds  of  data  elements:  pre-defined  and  collected.  Pre¬ 
defined  data  elements  must  be  defined  prior  to  using  the 
PAC.  Window  times  and  scripted  identifications  are  good 
examples  of  pre-defined  data.  Collected  data  elements  are 
captured  in  realtime  during  a  scenario  run.  Data  elements 
in  each  of  the  three  tables  are  listed  below.  Associated 
with  each  element  is  a  number.  This  number  is  used  to  index 
data  elements  to  columns  in  the  matrix  that  follows.  Note 
that  not  all  data  elements  are  used  to  calculate  measures. 
Some  are  collected  for  information  purposes  only. 


Threat  Group  Summary  Table 

1  Threat  Group  (pre-defined) 

2  Scripted  ID  (pre-defined) 

3  Threat  Group  Initial  Size  (pre-defined) 

4  First  ID  (pre-defined) 

5  ID  Window  End  (pre-defined) 

6  ID  Window  Size  (pre-defined,  information  only) 

7  Delta  Window  End  -  ID  (pre-defined,  information  only) 

8  First  Engage  (pre-defined) 

9  Engage  Window  End  (pre-defined) 

10  Engage  Window  Size  (pre-defined,  information  only) 

11  Delta  Window  End-  Engage  (pre-defined,  information  only) 

12  Number  of  Operator  IDs 

13  Number  of  System  IDs  (information  only) 

14  Time  of  Last  ID  (updated  after  every  ID  change) 

15  Last  ID  Assigned  (updated  after  every  ID  change) 

16  Source  of  Last  ID  (updated  after  every  ID  change) 

17  Time  to  ID  (Time  of  Last  ID  -  First  ID,  updated  after 
every  ID  change) 

18  Time  of  1st  Launch 

19  ATC  -  1st  Launch 

20  TLR  -  1st  Launch 
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Threat  Group  Summary  Table  (continued^ 

21  TTLL  -  1st  Launch 

22  Queue  Position  -  1st  Launch 

23  Time  of  Last  Launch  (information  only) 

24  Number  Early  Engages  (incremented  in  realtime  based  on 
comparison  of  launch  time  with  engagement  windows  for  the 
group) 

25  Number  Engages  within  Window  (incremented  in  realtime 
based  on  comparison  of  launch  time  with  engagement 
windows  for  the  group) 

26  Number  Late  Engages  (incremented  in  realtime  based  on 
comparison  of  launch  time  with  engagement  windows  for  the 
group) 

27  Sum  of  Engage  Delays  (a  running  total  updated  after  each 
launch;  an  engage  delay  is  time  of  launch  minus  First 
Engage;  each  engage  delay  value  is  added  to  the  current 
value  in  Sum  of  Engage  Delays) 

28  Number  of  Launches 

29  Low  Pk  Launches  (information  only) 

30  Number  of  Intercept  Failures 

31  Number  of  Intercept  Failures  Resulting  from  Low  Pk 
Launches 

32  Number  of  Group  Members  Killed 

33  Hook  Count 

34  Total  Duration  Hooks  (information  only) 

35  IFF  Count 

36  Time  of  Last  IFF  (information  only) 

37  Switch  Count 

38  Operator  Should  ID  Flag  (indicates  operator  should  ID 
this  group) 

39  Operator  Should  Engage  Flag  (indicates  operator  should 
engage  this  group) 


Asset  Penetration  File 

40  Threat  Group  (pre-def ined) 

41  Scripted  Group  Size  at  Penetration  (pre-def ined) 

42  Asset  ID  (pre-def ined) 

43  Asset  Value  (pre-def ined) 

44  Actual  Group  Size  at  Penetration 


Alert  Message  Table 
4  5  Bn 
4  6  Fp 

47  Console  number 

48  Group 

49  Time  Message  Was  Generated 

50  Time  Message  Was  Disposed 

51  Alert  Number 
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Alert  Message  Table  (continued! 

52  Sub-Alert  Number 

53  Alert  Text  (information  only) 

54  Disposal  Code  (dropped,  expired,  lost) 

55  Time  Acknowledged 
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Table  F-l 

PAC  Measures  by  PAC  Data  Element  Matrix 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 

Data  Elements 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 


Data  Elements 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 
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Table  F-l  Continued 

PAC  Measures  by  PAC  Data  Element  Matrix 
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APPENDIX  G 


PAC  Performance  Measure  Formulae 
Mission  Performance  Measures 


Defense  of  assets  — 

100  *  [1  -  (SUMi(SUMj (Penflgji  *  Assetvalue^ ) /SUMi ( j  * 

Assetvalue^) ) ] 

where, 

i  is  an  asset, 

j  are  tracks  scripted  to  penetrate  asset  i, 

Penflg  is  the  asset  penetration  flag.  Penflg  =  1  for  track  i 
against  asset  j  when  track  i  penetrates  asset  j.  Otherwise, 
Penflg  =  0. 

Assetvalue  =  9  -  the  priority  assigned  to  an  asset  (higher 
priority  assets  will  higher  Assetvalue  scores  under  this 
scheme) 


Attrition  =  100  *  (SUMh (Killf lgh) /h) 
where, 

h  =  the  number  of  threat  groups  scripted  as  hostiles, 
Killflg  =  1  for  hostiles  that  are  killed  and  0  for  hostiles 
that  survive. 


Friendly  Protection  =  100  *  [1  -  (SUMf (Kill f lgf ) /F) ] 
where , 

f  =  the  number  of  threat  groups  scripted  as  friends, 
Killflg  =  1  for  friends  that  are  killed  and  0  for  friends 
that  survive. 


Resource  Conservation  =  100  *  [  1  -  (Msl_waste/Msl_lnchs) ] 
where , 

Msl_waste  =  the  number  of  missiles  wasted  (a  wasted  missile 
is  a  missile  launched  at  a  friend  or  a  missile  launched  at  a 
hostile  with  TTFL  >  0  that  does  not  kill  the  hostile; 

Msl  lnchs  =  the  number  of  missiles  launched. 


Function  Performance  Measures  (FPMs) 

Friendly  Protector  FPMs; 

Note:  formulae  listed  for  the  Friendly  Protector  FPMs  are  for 
all  aircraft  in  a  scenario  (scripted  friends  and  hostiles) . 
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Formulae  for  friends  only  and  hostiles  only  are  similar, 
except  that  only  scripted  friends  or  scripted  hostiles  are 
considered . 


Percent  Tracks  IDed  =  100  *  (Op_ided_trks/trks__to_be_IDed) 
where, 

Op_ided_trks  =  number  of  tracks  (from  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator, 

Trks_to_be_IDed  =  the  number  of  tracks  to  be  IDed  by  the 
operator. 

Percent  Tracks  IDed  Correctly  = 

100  *  (Op_correct_IDs/Op_ided_trks) 
where, 

Op_correct_IDs  =  the  number  of  tracks  for  which  operator 
assigned  IDs  =  their  scripted  IDs 

Op_ided_trks  =  number  of  tracks  (from  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator. 

Percent  Tracks  Ided  Early  = 

100  *  (Early_Op_IDs/Op_ided_trks) , 
where, 

Early_Op_IDs  =  the  number  of  tracks  IDed  by  operator  that  are 
IDed  before  ID  window  start. 

Op_ided_trks  =  number  of  tracks  (from  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator. 

Percent  Tracks  Ided  Within  Window  = 

100  *  ( In_Window_Op_IDs/Op_ided_trks)  , 
where, 

In_Window_Op_IDs  =  the  number  of  tracks  IDed  by  operator  that 
are  IDed  after  ID  window  start  and  before  ID  window  end. 
Op_iued _trks  =  number  of  tracks  ffrom  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator. 

Percent  Tracks  Ided  Late  = 

100  *  (Late_Op_IDs/Op_ided_trks) , 
where, 

Late_Op_IDs  =  the  number  of  tracks  IDed  by  operator  that  are 
IDed  after  ID  window  end. 

Op_ided_trks  =  number  of  tracks  (from  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator. 

Average  Delay  to  ID  Tracks  = 

SUM (Op_ID_De lays ^ ) /Op_ided_trks , 
where, 

Op_ID_Delays  =  the  amount  of  time  in  seconds  from  beginning 
of  an  ID  window  to  when  the  track  is  IDed  by  the  operator. 
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Op_ided_trks  =  number  of  tracks  (from  tracks  to  be  IDed  by 
the  operator)  that  are  IDed  by  the  operator. 

%  Tracks  Neglected  = 

100  *  ( (Tracks_not_op_IDed  +  Tracks_IDed_incorrect  + 
Tracks_IDed_correct_but_late) /trks_to_be_IDed] , 
where , 

Tracks_not_op_IDed  =  number  of  tracks  (from  tracks  to  be  IDed 
by  the  operator)  that  are  not  IDed  by  the  operator, 
Tracks_IDed_incorrect  =  the  number  of  tracks  for  which 
operator  assigned  IDs  <>  their  scripted  IDs, 
Tracks_IDed_correct_but_late  =  the  number  of  tracks  IDed 
correctly  by  operator  but  after  ID  window  end, 

Trks_to_be_lDed  =  the  number  of  tracks  to  be  IDed  by  the 
operator. 

Weapons  Controller  FPMs : 

%  hostiles  engaged  = 

100  (number_engageable_hostiles_engaged  / 
number_engageable_hostiles) , where 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged, 

number_engageable_hostiles  =  the  number  of  scripted  hostiles 
that  are  flagged  as  being  engageable. 

%  hostiles  killed= 

100  (number_engageable_hostiles_killed  / 
number_engageable_host iles) , where 

number_engageable_hostiles_killed  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
killed, 

number_engageabJ e_hostiles  =  the  number  of  scripted  hostiles 
that  are  flagged  as  being  engageable. 

kill  ratio  =  100  *  (number_engageable_hostiles_killed/ 

number_engageable_hostiles_engaged) , 

where, 

number_engageable_hostiles_killed  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  av--> 
killed, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 

%  early  engagements  = 

100  * 

(number_engageable_hostiles_engaged_bef ore_vindow_start/ 
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number_engageable_hostiles_engaged) 

where, 

number_engageable_hostiles_engaged_before_window_start  =  the 
number  of  scripted  hostiles  that  are  engaged  before  window 
start, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged. 

%  engagements  within  window  = 

100  *  (number_engageable_hostiles_engaged_within_window/ 

number_engageable_hostiles_engaged) 

where, 

number_engageable_hostiles_engaged_within_window  =  the  number 
of  scripted  hostiles  that  are  engaged  at  or  after  window 
start  and  on  or  before  window  end, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 

%  engagements  late  = 

100  *  (number_engageable_hostiles_engaged_after_window_end/ 

number_engageable_hostiles_engaged) 

where, 

number_engageable_hostiles_engaged_after_window_end  =  the 
number  of  scripted  hostiles  that  are  engaged  after  window 
end, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 

average  delay  to  engage  = 

SUM (Op_Engage_Delays^) /number_engageable_hostiles_engaged , 
where , 

Op  Engage_Delays  =  the  amount  of  time  in  seconds  from 
beginning  of  an  engagement  window  to  when  the  track  is  first 
engaged  by  the  cperator. 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 

average  ATC  at  first  launch  = 

SUM (Op_Engage_ATCs^) /number_engageable_hostiles_engaged , 
where, 

Op_Engage_ATCs^  =  the  ATC  value  of  a  track  when  it  is  first 
engaged  by  the  operator, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged. 
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average  TLR  at  first  launch  = 

SUM (Op_Engage_TLR^) /number_engageable_host iles_engaged , 
where, 

Op_Engage_TLR^  =  the  TLR  value  for  a  track  when  it  is  first 
engaged  by  the  operator, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged. 

average  TTLL  at  first  launch  = 

SUM (Op_Engage_TTLL^) /number_engageable_hostiles_engaged , 
where, 

Op_Engage_TTLL^  =  the  TTLL  value  for  a  track  when  it  is  first 
engaged  by  the  operator, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 

average  queue  position  at  first  launch  = 

SUM (Op_Engage_queue ^ ) /number_engageable_hostiles_engaged , 
where, 

Op_Engage_queue^  =  the  position  in  engage  data  queue  for  a 
track  when  it  is  first  engaged  by  the  operator, 
number_engageable_host i les_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged. 

%  missile  wastes  due  to  weapons  controller  =  100  * 

( intercept_failures_for_unauthorized_low_PK_hostile_engages/ 

total_wastes) 

where, 

intercept_failures_for_unauthorized_low_PK_hostile_engages  = 
the  number  of  intercept  failures  occurring  for  launches  with 
TLR  >  0  and  low  PK  engagements  not  authorized, 
total_wastes  =  number  of  engagements  at  scripted  friends  plus 
intercept_failures_for_unauthorized_low_PK_hostile_engages . 

average  number  of  launches  per  track  engaged  = 
total_hostile_launches/number_engageable_hostiles_engaged , 
where, 

total_hostile_launches  =  the  total  number  of  launches  against 
hostile  tracks, 

number_engageable_hostiles_engaged  =  the  number  of  scripted 
hostiles  that  are  flagged  as  being  engageable  that  are 
engaged . 
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Task  Performance  Measures 
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%  all  tracks  IFFed  =  100  * 

(number_tracks_IFFed/ tot al_number_t racks) , 
where, 

number_tracks_IFFed  =  number  of  tracks  with  IFF  count  >  0, 
total_number_tracks  =  number  of  tracks  in  the  scenario. 

%  scripted  friends  IFFed  =  100  * 

(number_scripted_f riends_I FFed/ number _scripted_f riends) , 
where, 

number_scripted_f riends_IFFed  =  number  of  scripted  friends 
with  IFF  count  >  0, 

total_number_t racks  =  number  of  scripted  friends  in  the 
scenario. 

%  scripted  hostiles  IFFed  =  100  * 

(number_scripted_hostiles_IFFed/number__scripted_hostiles)  , 
where , 

number_scripted_hostiles_IFFed  =  number  of  scripted  hostiles 
with  IFF  count  >  0, 

total_number_tracks  =  number  of  scripted  hostiles  in  the 
scenario . 

%  all  tracks  hooked  =  100  * 
(number_tracks_hooked/total_number_tracks) , 
where, 

number_tracks_hooked  =  number  of  tracks  with  hook  count  >  0, 
total_number_tracks  =  number  of  tracks  in  the  scenario. 

%  scripted  friends  hooked  =  100  * 

(number_scripted_friends_hooked/number_scripted_f riends) , 
where, 

number_scripted_friends_hooked  =  number  of  scripted  friends 
with  hook  count  >  0, 

total_number_tracks  =  number  of  scripted  friends  in  the 
scenario. 

%  scripted  hostiles  hooked  =  100  * 

(nuirber_scripted_host iles_hooked/number_scripted_hostiles)  , 
where, 

number_scripted_hostiles_hooked  =  number  of  scripted  hostiles 
with  hook  count  >  0, 

total_number_tracks  =  number  of  scripted  hostiles  in  the 
scenario . 

average  number  switch  actions  per  track  = 
SUM(track-related_switch_actions^) /number  of  tracks 
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where , 

track-related_switch_actions^  =  the  number  of  engage  and  ID 
switch  actions  made  for  each  track  i  in  the  scenario, 
number  of  tracks  =  the  total  number  of  tracks  in  the  scenario 
that  the  operator  had  to  act  upon. 

%  alerts  lost  =  100  * 

(number_alerts_lost/number_alerts_generated) , 
where , 

number_alerts_lost  =  total  number  of  alerts  that  are  lost 
(are  dropped  from  the  queue  because  all  slots  are  filled) 
number_alerts_generated  =  total  number  of  alerts  that  are 
generated. 

%  alerts  expired  =  100  * 

(number_alerts_expired/number_alerts_generated) , 
where , 

number_alerts_expired  =  total  number  of  alerts  that  expire 
(referenced  track  is  dropped  before  alert  makes  it  to  AML) , 
number_alerts_generated  =  total  number  of  alerts  that  are 
generated . 

%  alerts  displayed  =  100  * 

(number_alerts_displayed/number_alerts_generated) , 
where, 

number_alerts_displayed  =  total  number  of  alerts  that  get 
displayed  on  the  alert  message  line  (AML) , 
number_alerts_generated  =  total  number  of  alerts  that  are 
generated . 

average  delay  to  display  an  alert  = 

SUM ( t ime_to_display_alert ^ ) /total_number_alerts_displayed , 
where , 

time_to_display_alert^  =  time  from  generation  of  alert  i 
until  it  is  displayed  on  AML, 

total_number_alerts_displayed  =  total  number  of  alerts  that 
get  displayed  on  AML. 

average  delay  to  acknowledge  an  alert= 

SUM ( time_to_acknowledge_alert^) /total_#_alerts_acknowledged , 
where, 

time_to_acknowledge_alert^  =  time  from  display  of  alert  i 
until  it  is  acknowledged, 

total_#_alerts_acknowledged  =  total  number  of  alerts  that  get 
acknowledged . 
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